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Executive  Summary 


Federal  agencies'  annual  performance  reports  should  give  Congress  and  the  public  accurate,  timely  information 
documenting  the  tangible  public  benefits  the  agencies  produce.  The  Bush  administration  has  indicated  that  it 
will  begin  using  agency  performance  information  later  this  year  when  making  budget  decisions  for  fiscal  year 
2003.  Researchers  at  the  Meieatus  Center  conducted  our  second  annual  evaluation  of  the  reports  produced  by 
the  24  agencies  covered  under  the  Chief  Financial  Officers  Act,  employing  the  same  criteria  to  evaluate  the  fiscal 
year  2000  performance  reports  that  we  employed  to  evaluate  the  fiscal  year  1999  reports.  Our  scorecard  contin¬ 
ues  to  address  three  questions: 

•  Does  the  agency  report  its  accomplishments  in  a  transparent  fashion  that  makes  accomplishments  and 
problems  clear? 

•  Does  the  report  focus  on  documenting  tangible  public  benefits  the  agency"  produced? 

•  Does  the  report  show  evidence  of  forward-looking  leadership  that  uses  performance  information  to 
devise  strategies  for  improvement? 

By  assessing  the  quality^  of  agency  reports,  we  seek  to  ascertain  which  agencies  are  supplying  the  information 
that  Congress  and  the  public  need  to  make  informed  funding  and  policy  decisions. 

BEST  REPORTS:  For  fiscal  2000,  the  Department  of  Veterans  Affairs  performance  report  scored  highest,  fol¬ 
lowed  closely  by  the  Department  of  Transportation.  The  fiscal  2000  report  of  the  U.S.  Agency  for  International 
Development  ranked  third,  down  from  first  place  in  fiscal  1999. 

REPORTS  MOST  IN  NEED  OF  IMPROVEMENT:  Of  the  23  agencies  that  released  reports  by  April  27,  the 
National  Aeronautics  and  Space  Administration's  report  scored  lowest.  The  Department  of  Health  and  Human 
Services  and  Nuclear  Regulatory"  Commission  rounded  out  the  bottom  three.  The  Department  of  Agriculture 
released  its  repc')!*!  on  May  1,  more  than  a  month  after  the  statu toiy  deadline  and  too  late  to  be  included  in  our 
analysis. 

MOST  IMPROVED  REPORTS;  The  National  Science  Foundation  exhibited  the  most  improvement  since  last 
year,  jumping  from  last  place  in  fiscal  1999  to  sixth  in  fiscal  2000.  The  Department  of  Justice  showed  the  next 
largest  improvement,  rising  from  21st  place  to  fifth. 

MOST  COMMON  STRENGTHS:  (1)  Readability  of  the  reports,  (2)  dear  articulation  of  results-based  goals, 
and  (3)  discussion  of  major  management  challenges. 

MOST  COMMON  WEAKNESSES:  (1)  Failure  to  make  reports  accessible  to  the  public,  (2)  failure  to  demon¬ 
strate  a  cause  and  effect  relationship  between  the  agency^s  action  and  observed  outcomes,  and  (3)  failure  to  link 
performance  data  to  costs.  Notwithstanding  these  common  w'eaknesses,  a  number  of  agencies  did  well  on  these 
three  attiibutes. 

MODEST  BUT  WIDESPREAD  IMPROVEMENT:  The  average  score  was  approximately  5  percent  higher  for 
the  fiscal  2000  reports  than  for  fiscal  1999  reports,  in  spite  of  the  fact  that  scoring  was  more  stringent.  More  than 
half  of  the  reports  scored  higher  in  fiscal  2000  than  in  fiscal  1999. 


Scorecard  Summary  and  Ranking  for  Fiscal  2000 


^  RANKING  IS  FROM  HIGHEST  TO  LOWEST  TOTAL  SCORE. 
MAXIMUM  POSSIBLE  SCORE  IS  60. 
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Comparison  of  Fiscal  1999  and  Fiscal  2000 
Scores  and  Rankings 
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Introduction 


For  several  years,  federal  agencies  have  developed  strategic  plans,  performance  plans,  and  (most  recently)  per¬ 
formance  reports  to  explain  what  they  are  trying  to  accomplish,  identify  performance  measures,  and  report  on 
their  results.  The  upcoming  fiscal  year  will  see  the  next  logical  step  as  policymiakers  start  to  take  account  of  per¬ 
formance  information  when  they  make  budget  decisions. 


President  Bush's  fiscal  year  2002  budget  proposal  calls  upon  the  federal  government  to  produce  better  results  for 
citizens  by  enhancing  accountability  for  dollars  raised  and  dollars  spent.  Later  this  year,  the  administi'ation  will 
begin  using  information  on  agency  performance  to  establish  fiscal  year  2003  budgets  for  a  selected  set  of  pro¬ 
grams.^  Performance-based  budgeting  means  that  money  will  be  allocated  not  just  on  the  basis  of  perceived 
needs  and  policy  priorities,  but  also  according  to  the  federal  governments  ability  to  effectively  address  those 
needs  and  priorities.  A  desire  to  accomplish  a  goal  or  solve  a  problem  will  no  longer  be  enough;  program  pro¬ 
ponents  will  have  to  demonstrate  that  the  programs  they  advocate  can  actually  accomplish  their  goals. 

For  performance  budgeting  to  work,  performance  information  must  be  transparent,  accessible,  and  reliable.  The 
Government  Performance  and  Results  Act  requires  federal  agencies  to  produce  annual  performance  reports. 

The  purpose  of  these  reports  is  to  give  Congress  and  the  American  people  accurate,  timely  information  that  will 
let  them  assess  the  extent  to  which  agencies  are  producing  tangible  public  benefits.  In  line  with  expectations 
under  the  legislation,  agencies  published  their  first  reports  (for  fiscal  year  1999)  in  March  2000.  Their  reports  for 
fiscal  year  2000  were  due  on  March  31,  2001  (April  9  if  combined  with  the  fiscal  year  2002  Annual  Performance 
Plan)." 

While  the  goal  of  public  agencies  is  community  benefit  rather  than  private  profit,  government  performance 
reports  should  meet  or  exceed  the  standards  for  transparency  and  disclosure  established  by  the  best  annual 
reports  produced  by  publicly-held  corporations.  A  w^ell-crafied  corporate  annual  report  provides  a  comprehen¬ 
sive,  concise,  accurate,  and  reliable  assessment  of  the  company's  operating  and  financial  performance.  Effective 
accountability  in  public  service  requires  that  government  present  a  comprehensive,  concise,  accurate,  and  reli¬ 
able  assessment  of  the  benefits  it  created  for  the  public  and  the  full  cost  of  producing  those  benefits.  Equipped 
wuth  such  information,  the  administration  and  Congress  can  allocate  federal  resources  in  w^ays  that  continually 
advances  government's  contribution  to  citizens'  quality  of  life.^ 


^Office  of  Management  and  Budget,  Budget  for  Fiscal  Year  2002-1.  Improving  Government  Performance,  available  at 
http://w^ww.whitehouse.gov/omb/budget/fy2002/bud01.html. 

“Tl'ie  Mercatus  Center  has  developed  a  seven-step  process,  called  "Outcome-Based  Scrutiny,"  that  provides  a  framework  for 
comparing  the  results  and  costs  of  programs  with  similar  objectives  and  assessing  the  likely  impact  of  reallocating  resources 
to  the  most  effective  programs.  For  a  pilot  study  applying  Outcome-Based  Scrutiny  to  federal  vocational  training  programs, 
see  ww' w.mercatus.org/obs/ 


2ND  Annual  Performance  Report  Scorecard 


I 


Mercatus  Center  •  May  2001 


To  help  policymakers  assess  this  year's  reports  and  help  agencies  improve  the  quality''  of  next  year  s  reports,  an 
eight-member  research  team  at  the  Mercatus  Center  evaluated  the  reports  produced  by  the  24  agencies  covered 
under  the  Chief  Financial  Officers'  Act,  This  marks  the  second  year  that  researchers  at  the  Mercatus  Center  s 
Government  Accountability  Project  have  evaluated  agency  performance  reports.  Our  first  Pevfonnuyice  Report 
Scorecard,  published  in  May  2000,  generated  a  large  number  of  inquiries  from  both  the  public  and  the  federal 
government.^  We  hope  this  annual  assessment  will  help  inform  decisionmakers  and  promote  the  American 
spirit  of  competition  in  government  performance  reporting. 


Evaluation  Criteria 


The  Mercatus  research  team  employed  12  criteria  to  address  three  questions: 

•  Does  the  agency  report  its  accomplishments  in  a  transparent  fashion? 

•  Does  the  report  focus  on  documenting  tangible  public  benefits  the  agency  produced? 

•  Does  the  report  show  evidence  of  forward-looking  leadership  that  uses  performance  information  to 
devise  strategies  for  improvement? 

The  purpose  of  this  assessment  is  not  to  evaluate  the  quality  of  the  actual  results  the  agencies  produced.  Our 
goal  is  to  ascertain  how  well  the  agency  reports  inform  the  public  about  the  results  they  produced.-' 


"Of  the  24  CFO  Act  agencies,  8  contacted  us  to  arrange  followup  meetings  to  discuss  our  evaluation  of  their  reports. 

The  scorecard  for  agencies'  1999  performance  reports  is  available  at  www.mercatus.org/scorecard/ 

^The  General  Accounting  Office  evaluated  the  quality  of  agencies'  results  after  the  fiscal  1999  reports  were  published  and  will 
do  so  again  for  fiscal  2000. 
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Evaluation  Criteria 


TRANSPARENCY  ^ _ 

1.  Is  the  report  easily  accessible  and  easily  identified  as  the  agency's  Annual  Performance  Report? 

2.  Is  the  report  easy  for  a  layperson  to  read  and  understand? 

3.  Are  the  performance  data  reliable,  credible,  and  verifiable? 

4.  Did  the  agency  provide  baseline  and  trend  data  to  put  its  performance  measures  in  context? 


PUBLIC  BENEFITS  ^ _ 

5.  Are  the  goals  and  objectives  stated  as  results? 

6.  Are  the  performance  measures  focused  on  results  or  activities? 

7.  Does  the  agency  demonstrate  that  its  actions  have  actually  made  a  significant  contribution  toward  its 
stated  goals? 

8.  Did  the  agency  link  its  goals  and  results  to  costs? 


LEADERSHIP 


9.  Does  the  report  show  how^  the  agency's  results  will  make  this  country  a  better  place  to  live? 

10.  Does  the  agency  explain  failures  to  achieve  targets  outlined  in  its  strategic  plan? 

11.  Does  the  report  identify  major  management  challenges? 

12.  Does  it  describe  changes  in  policies  or  procedures  to  do  better  next  year? 
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TRANSPARENCY 


Inherent  in  the  Government  Performance  and  Results  Act's  reporting  requirements  is  the  principle  that  tire 
reports  should  be  accessible,  readable,  and  useable  by  a  wide  variety  of  audiences,  including  Congress,  the  gen¬ 
eral  public,  the  news  media,  stakeholders,  and  interest  groups.  In  the  absence  of  a  report  that  clearly  indicates 
achievements  and  problems,  an  agency's  benefits  to  the  community  will  remain  secret  to  all  but  a  few  insiders, 
and  budget  decisions  will  fail  to  reflect  realistic  assessments  of  what  agencies  can  accomplish. 

PUBLIC  BENEFITS 

The  clear  intent  of  the  Government  Performance  and  Results  Act  is  to  focus  on  meaningful  results  produced  for 
the  community.  In  the  absence  of  information  about  public  benefits,  budgets  reflect  decisionmakers  wishes 
rather  than  a  sober  assessment  of  the  government's  ability  to  make  those  wishes  come  true.  To  clearly  demon¬ 
strate  an  agency's  value  to  the  public,  each  report  vShould  enunciate  goals  and  measures  focused  on  actual 
results.  A  result  is  the  benefit  produced  or  harm  avoided  for  a  particular  set  of  clients  or  the  public  at  large.  In 
the  past,  goals  and  measures  often  emphasized  day-to-day  activities  and  procedures,  such  as  levels  of  effort, 
expenditures,  or  internal  project  milestones.  This  reporting  practice  implicitly  assumed  that  the  amount  of  effort 
spent  on  activities  automatically  translates  into  results.  Such  an  assumption  can  be  incorrect  for  a  wide  variety 
of  reasons.  If  the  reporting  process  does  not  review  actual  results  achieved,  then  it  does  not  fulfill  the  intent  of 
the  Government  Performance  and  Results  Act  to  inform  the  public  of  the  success  or  failure  of  government  pro¬ 
grams. 

FORWARD-LOOKING  LEADERSHIP 

Each  report  ought  to  identify  and  offer  solutions  to  major  management  challenges  by  putting  forth  a  plan  to 
improve  policies  and  procedures  in  the  next  reporting  cycle.  Forward-looking  leadership  means  that  the  agency 
uses  the  information  in  the  performance  report  to  identify  solutions  to  perceived  problems  and  change  future 
plans  to  capitalize  on  opportunities  for  improvement. 

INTERPRETING  OUR  FINDINGS 

The  research  team  evaluated  the  qualih/  of  reporting,  not  the  ipinliti/  of  results,  so  it  would  be  a  mistake  to  conclude 
that  the  agencies  with  the  highest-scoring  reports  necessarily  produced  the  best  results  for  the  community. 
Ideally,  an  agency's  report  reflects  more  about  its  managers'  capabilities  than  just  their  ability  to  write  leports.  It 
would  also  be  inappropriate  to  draw  policy  conclusions  from  our  inquiry;  we  offer  no  recormnendations  on 
whether  the  federal  government  should  or  should  not  be  engaged  in  its  current  menu  of  activities. 

So  what  do  the  findings  in  this  study  really  mean?  By  assessing  the  quality  of  agency  reports,  we  seek  to  ascer¬ 
tain  which  agencies  are  supplying  the  information  that  Congress  and  the  public  need  to  make  informed  funding, 
budgeting  and  policy  decisions. 

An  additional  word  on  information  quality  is  also  in  order.  The  researchers  assessed  the  quality  of  each  repoit  s 
disclosure  of  verification  and  validation  procedures,  but  in  the  interest  of  producing  a  timely  study,  we  did  not 
independently  verify  the  performance  information  in  each  agency's  report.  Given  the  importance  of  accurate 
data  for  sensible  decisions,  we  believe  that  verification  and  validation  should  be  a  high  priority  for  agency 
Inspectors  General,  Congi'ess,  the  General  Accounting  Office,  and  the  Office  of  Management  and  Budget. 
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Scoring  the  Reports:  Role  Models  and  Bad  Influences 


Each  agency  had  the  opportunity  to  earn  up  to  20  points  in  each  of  the  three  categories,  for  a  maximum  score  of 
60  points.  Each  category  included  four  equally  weighted  evaluation  factors. 


Tranparency 


1.  Is  the  report  easily  accessible  and  easily  identified  as  the  agency's  Annual  Performance  Report? 

Public  accountability  can  only  be  served  if  members  of  the  public  can  actually  find  out  what  the  agency  is  doing 
for  them.  The  annual  report  should  be  made  accessible  to  the  public,  stakeholders,  the  media,  and  others  at  the 
agency's  web  site.  The  report  should  be  filed  as  a  stand-alone  document  and  linked  to  the  agency  s  home  page 
-  not  buried  in  a  monstrous  document  or  rarely  visited  web  site.  Reports  for  fiscal  year  2000  were  due  on  March 
31,  2001,  or  on  April  9  if  combined  with  the  fiscal  year  2002  Annual  Performance  Plan. 

An  agency's  score  on  this  criterion  depended  chiefly  on  whether  the  report  was  available  on  the  agency  s  web 
site,  how  difficult  it  was  for  us  to  find  it,  whether  the  report  was  clearly  labeled  as  the  fiscal  year  2000  perform¬ 
ance  report,  and  whether  it  was  broken  into  clearly-labeled  sections  for  convenient  downloading. 

Role  models:  Deparhneni  of  Labor  (DOL)  and  Environmental  Protection  Agencp  (EPA), 

Featured  on  the  home  page,  DOL's  fiscal  2000  Annual  Report  can't  be  missed.  The  link  has  a  nice  graphic,  and  a 
few  sentences  describe  the  contents.  An  index  offers  about  two  dozen  separate  sections  for  Portable  Document 
Format  (pdf)  download  -  all  <500KB. 

EPA' s  report  was  online  at  the  beginning  of  March,  with  easy-fo-find  links  on  EPA' s  home  page,  clear  explana¬ 
tion  of  the  combination  of  this  report  with  the  Accountability  Report  and  convenient  separation  into  sections  for 
downloading. 

Bad  influences:  As  of  April  27,  nine  agencies  had  not  i/et  made  their  reports  available  on  their  web  pages, 

2.  Is  the  report  easy  for  a  layperson  to  read  and  understand? 

Style  and  language  should  allow  a  non-specialist  to  grasp  significant  information  quickly.  It  should  read  more 
like  a  corporate  annual  report  to  a  wide  range  of  readers  than  an  internal  memo  for  management.  Key  elements 
for  scoring  purposes  include  clarity  of  text,  absence  of  jargon  and  acronyms,  sentence  and  paragraph  structure, 
general  organization,  and  effective  use  of  visual  features  like  graphics,  tables,  and  headings. 

Role  models:  Department  of  Energy  (DOE)  and  Department  of  the  Interior  (DOI). 

Both  reports  are  well-structured,  clear  and  easy  to  read.  Acronyms  are  defined.  They  also  use  attractive  charts 
to  aid  understanding.  DOE  makes  excellent  use  of  photographs  to  bring  its  story  to  life. 
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Bad  influences:  Department  of  Education  (Education)  and  National  Aeronautics  and  Space  Administration  (NASA). 

Education's  report  is  almost  impossible  to  decipher.  First  of  all,  its  organization  is  very  confusing;  two  volumes 
are  not  integrated  and  their  relationship  is  not  made  clear.  It  is  an  "interim"  report  with  a  text  box  on  the  cover 
warning  that  it  does  not  contain  the  (intended)  Fiscal  Year  2002  Performance  Plan  information  because  of  the 
president's  pending  proposal  that  evidently  has  large  department-wide  policy  implications.  That  s  understand¬ 
able.  This  label  misleads  readers,  though,  because  one  finds  upon  closer  inspection  that  the  report  is  also  miss¬ 
ing  most  fiscal  2000  results.  A  thorough  explanation  of  the  circumstances  and  a  presentation  limited  to  actual 
data  that  are  available  would  be  a  more  honest  and  considerate  approach. 

The  organization  of  NASA's  report  is  confusing,  as  it  attempts  to  discuss  both  "enterprises"  and  "cross-cutting 
processes."  The  summary  tables  are  hard  to  decipher,  technical  language  and  acronyms  are  excessive,  measures 
are  too  numerous  and  editing  is  poor. 

Even  attempts  to  improve  the  overall  feel  of  the  report  with  photos  are  ineffective  because  captions  fail  to 
explain  the  images.  For  example,  an  appealing  picture  of  Earth  has  the  caption  "Global  vegetation  index  map 
acquired  by  MODIS  instrument."  There  is  no  reference  to  the  associated  objective  or  performance  target.  On 
close  inspection,  the  reader  can  find  the  acronym  definition  (Moderate  Resolution  Imaging  Spectroradiometer). 
The  target  is  "Continue  the  development  of  a  global  land  cover/ use  change  data  set  based  on  Landsat  and  EOS 
instrument,  at  seasonal  refresh  rate."  The  objective  is  "Understand  the  causes  and  consequences  of  land 
cover/land  use  change."^ 

3.  Are  the  performance  data  reliable,  credible,  and  verifiable? 

The  report  should  discuss  data  quality  and  verification/ validation  procedures  and  should  provide  a  summary 
of  the  Inspector  General's  report.  Outside  audits  are  desirable.  People  outside  the  agency  should  be  able  to 
access  and  check  the  data. 

Role  models:  Department  of  Transportation  (DOT)  and  National  Science  Foundation  (NSF). 

DOT'S  data  are  selected,  presented  and  discussed  in  a  convincing  way.  Furthermore,  an  extensive  appendix 
describes  their  scope,  source,  limitations  and  verification/ validation  procedures  in  a  very  thorough  fashion. 

NSF  discusses  in  detail  its  procedures  for  collecting  and  assessing  data.  NSF  hired  a  private  consulting  firm 
(PricewaterhouseCoopers)  to  audit  its  performance  measurement  process;  in  PwC's  opinion,  it  is  valid  and  ver¬ 
ifiable."  NSF's  Inspector  General  notes  that  data  verification  is  a  problem,  but  NSF  seems  to  be  taking  a  lot  of 
steps  to  address  it. 

Bad  influences:  Some  agencies  rely  solely  on  internal  data  sources  and  fail  to  discuss  data  validity  and  verifica¬ 
tion.  These  include  the  Department  of  Justice,  Department  of  State,  and  Social  Securih/  Admin ish  ation.  The  Federal 
Emergency  Management  Agency  apparently  has  significant  data  problems,  but  the  details  are  a  mystery.  A  full 
paragraph  of  the  three-paragraph  director's  letter  apologizes  for  FEMA's  data.  Such  candor  is  praiseworthy,  but 
data  quality  and  sources  are  not  discussed  elsewhere  in  the  report. 


'See  p.  25  of  NASA's  report. 
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4.  Did  the  agency  provide  baseline  and  trend  data  to  put  its  performance  measures  in  context? 

Performance  information  is  meaningless  unless  it  is  easily  tracked  over  several  years.  To  assign  scores  under 
this  criterion,  vve  considered  both  the  quantity  and  presentation  of  baseline  and  trend  data. 

Some  measures  have  been  changed  since  FY  1.999  and  therefore  lack  baselines.  In  some  cases,  proper  measures 
have  just  been  identified  and  systems  have  not  been  established  to  provide  the  necessary  data.  These  circum¬ 
stances  should  be  explicitly  stated  in  the  report. 

Role  models:  Departuient  of  Transportation  (DOT)  and  Department  of  Justice  (DOJ). 

DOT  frequently  provides  as  much  as  10  years  of  data  that  assist  understanding  not  only  of  trends  but  of  target 
selection  as  well.  In  several  cases,  additional  information  is  supplied  that  adds  meaning,  such  as  sample  size  (if 
the  target  is  a  percentage),  a  breakdown  of  causal  factors,  and  classification  by  mode  of  travel. 

DOJ's  graphical  presentation  of  data  reveals  both  trends  and  projections.  Design  and  labeling  is  unifoim  and 
helpful. 

Bad  influences:  Nuclear  Regulaton/  Commission  (NEC)  and  NASA, 

NRC  presents  hardly  any  meaningful  data,  despite  a  wealth  of  quantified  data  that  was  surely  reported  by 
licensees  for  many  years  prior  to  1999.  NRC  seems  to  aim  for  mere  compliance  with  the  Office  of  Management 
and  Budget's  guidelines. 

Most  of  NASA's  performance  measures  are  detailed  activities,  making  year-to-year  comparison  impossible. 
Tiend  Data  tables  fail  to  present  these  activity'  lists  in  a  manner  that  would  indicate  trends.  A  column  labeled 
"Trend  Assessment"  actually  says  little  about  trends.  Most  targets  are  new,  completed  or  eliminated;  for  those 
remaining,  we  get  a  status  report. 


Benefits  to  the  Community 


5*  Are  the  goals  and  objectives  stated  as  results? 

A  "result  goal"  is  defined  as  the  intended  benefit  (or  harm  avoided)  due  to  the  agency's  programs  or  activities. 
All  goals  should  also  clearly  be  of  value  to  the  community.  Activity  measures,  such  as  number  of  participants  in 
a  particular  program  or  completion  of  a  study,  may  contribute  to  achievement  of  a  result  but  do  not  constitute 
results  of  interest  to  the  public  at  large. 

Role  models:  Department  of  Veterans  Affairs  and  Department  of  Housing  and  Urban  Development. 

Virtually  all  of  these  agencies'  strategic  goals  and  objectives  focus  on  results.  They  are  concise  and  should  lead 
to  relatively  easy  establishment  of  performance  measures. 
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Department  of  Veterans  Affairs  Strategic  Goals 


Strategic  Goal  1: 


Strategic  Goal  2: 
Strategic  Goal  3: 


Strategic  Goal  4: 


Restore  the  capability  of  disabled  veterans  to  the  greatest  extent  possible  and  improve 
the  quality  of  their  lives  and  that  of  their  families 

Ensure  a  smooth  transition  for  veterans  from  active  military  service  to  civilian  life 

Honor  and  serve  veterans  in  life  and  memorialize  them  in  death  for  their  sacrifices  on 
behalf  of  the  Nation 

Contribute  to  the  public  health,  socioeconomic  well  being  and  history  of  the  Nation 


Department  of  Housing  and  Urban  Development  Strategic  Goals 

Strategic  Goal  1:  Increase  the  availability  of  decent,  safe  and  affordable  housing  in  American 
communities 


Strategic  Goal  2: 
Strategic  Goal  3: 

Strategic  Goal  4: 
Strategic  Goal  5: 

V 


Ensure  equal  opportunity  in  housing  for  all  Americans 

Promote  housing  stability,  self-sufficiency  and  asset  development  of  families  and 
individuals 

Improve  community  quality  of  life  and  economic  vitality 
Ensure  public  trust  in  HUD 


j 


Bad  influence:  Office  of  Personnel  Management, 


Strategic  goals  are  clearly  activities  and  read  more  like  policy  statements  whose  outcomes  are  vague. 

Goal  II:  Protect  and  promote  the  merit-based  Civil  Service  and  the  employee  earned  benefit  programs 
through  an  effective  oversight  and  evaluation  program 

Goal  III:  Provide  advice  and  assistance  to  help  Federal  agencies  improve  their  human  resources 
management  programs  to  effectively  operate  within  the  economy,  demographics  and 
environment  of  the  21.st  Century^ 

Agencies  often  offer  a  mixture  of  activity  and  result  goals.  It  is  not  clear  whether  activity  goals  are  offered  out  of 
desperation  to  find  something  measurable,  or  because  there  is  confusion  surrounding  the  difference  between 
activities  and  results. 
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6.  Are  the  performance  measures  focused  on  results  or  activities? 


Direct  measures,  surrogate  measures  and  indicators  should  tell  whether  or  not  intended  results  were  achieved. 
For  instance,  if  the  goal  is  to  reduce  suffering  from  disease,  a  direct  measure  would  be  a  detailed  accounting  of 
costs  in  time,  money  and  '^quality  of  life/^  a  surrogate  measure  might  be  a  nationwide  study  that  included 
physicians'  assessments  of  severity;  and  an  indicator  would  be  incidence  of  the  disease.  None  are  perfect  but  all 
deal  with  results. 

Data  measuring  activities,  funding,  or  effort  do  not  usually  qualify  as  results,  unless  the  agency  makes  a 
compelling  and  documented  case  that  there  is  a  tight  link  between  activity’^  and  achievement  of  results.  For  the 
disease  example,  inappropriate  measures  would  include  the  number  of  treatment  centers,  dollars  spent  on 
treatment  or  number  of  hotline  phone  calls.  All  of  these  may'  contribute  to  the  result  but  do  not  in  and  of 
themselves  show  a  reduction  of  suffering  from  disease. 

Role  model:  Department  ofTnmsportation  (DOT)  and  US.  Agency  for  International  Development  (USAID). 

The  vast  majority^  of  DOT's  measures  indicate  resxilts  of  interest  to  citizens  rather  than  activities  or  outputs. 
Result-based  measures  include: 

•  Percent  of  runways  in  good  or  fair  condition 

•  Average  condition  of  rail  vehicle  fleet 

•  Percent  of  days  in  shipping  season  that  locks  are  available  in  waterways 

•  Hours  of  delay  per  1000  vehicle-miles-traveled  on  federal-aid  highways 

USAID's  measures  are  all  results.  The  only  problem  is  that  they  measure  countries'  or  regions'  results,  not  the 
agency''s  results.  The  agency’  acknowledges  the  difference  and  makes  a  commendable  effort  to  explain  the  link 
between  its  activities  and  results  in  particular  countries. 

Bad  influence:  NASA, 

Targets  are  extremely  activity^-based.  Agency  targets  are  "things  to  do"  that  are  so  specific,  they  must  be 
discarded  when  accomplished  and  replaced  with  new  things  that  are  entirely  different.  While  the  activities  may 
be  necessary'  and  logical,  their  documentation  is  meaningless  to  outsiders.  For  example,  one  goal  for  space 
science  is  "Demonstrate  performance  of  the  Superconductor-Insula tor-Superconductor  (STS)  mixer  to  at  least 
8hv/k  at  1,120  GHz  and  lOhv/k  at  1,200  GHz."  This  may  be  an  important  operational  goal,  but  the  link  to 
results  is  unclear. 

7.  Does  the  agency  demonstrate  that  its  actions  have  actually  made  a  significant  contribution  toward  its 
stated  goals? 

Does  the  report  make  a  case  that  there  is  a  clear  cause/  effect  relationship  between  the  agency  s  work  and  results 
valued  by  the  public?  It  should  be  obvious  to  the  layman  that  the  agency's  programs  influence  results  in  a 
positive  way.  Additionally^  the  agency  should  not  take  credit  for  other  positive  influences.  For  example,  if 
traffic  fatalities  fell  because  eveiyone  decided  to  stay  home,  then  the  Department  of  Transportation  would  not 
deserve  credit  for  the  drop  in  fatalities. 
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Role  model:  General  Services  Administration  (GSAj, 


Because  of  the  nature  of  GSA's  role  in  the  government,  which  is  to  serve  other  agencies'  business  needs, 
improving  its  own  processes  often  automatically  benefits  its  customers  in  the  federal  government  and 
presumably  citizens/ taxpayers.  The  connection  between  what  GSA  does  and  the  expected  result  is  obvious. 

This  report  gives  tangible  evidence  of  savings  to  its  customers.  The  stor}*'  isn't  told  at  the  highest  level,  but  at 
the  performance  goal  level,  impact  is  clear. 

Bad  influences:  Depmrtmeni  of  Health  and  Hiinum  Services  (HHS). 

Although  HHS  enunciates  very  good  results-based  goals,  its  report  demonstrates  no  causal  link  between  HHS 
activities  and  results.  Anecdotes  and  selected  data  points  tell  no  story  at  all  about  performance  against 
objectives,  and  the  department  offers  no  conclusions  in  any  major  area  of  concern. 

8.  Did  the  agency  link  its  goals  and  results  to  costs? 

Performance  budgeting  will  begin  with  fiscal  2003.  We  included  this  factor  in  order  to  give  credit  to  agencies 
that  took  the  initiative  to  include  cost  information  in  highly  useful  and  informative  ways.  For  example,  an 
agency^  could  show  the  quantity  of  funding  or  other  resources  that  are  allocated  to  each  of  its  major  i‘esults~based 
goals,  or  it  might  report  the  cost  per  unit  of  some  more  specific  performance  measure. 

Role  models:  Department  of  Veterans  Affairs  (VAf  Small  Business  Administration  (SBAf  and  Department  of  Energy 
(DOE). 

Various  sections  in  the  VA  report  show  spending  by  program,  organization,  and  strategic  goal.  VA  is  far  ahead 
of  most  other  agencies  in  reporting  cost  information. 

SBA  does  an  impressive  job  of  linking  costs  with  results;  it  identifies  costs  at  the  performance  indicator  level  for 
FY  2000.  All  that  is  missing  is  a  summary  of  costs  at  higher  levels,  and  perhaps  some  analysis. 

DOE  shows  net  costs  for  each  "Decision  Unit,"  Because  of  goal  overlaps,  the  cost/benefit  relationships  are  not 
very  clear.  Nevertheless,  the  amount  of  resource  information  and  progress  in  mapping  out  a  performance 
budgeting  scheme  is  quite  impressive. 


Bad  influences:  Department  of  State,  Federal  Emergency  Management  Agency  (FEMA),  and  Department  of  Health  and 
Human  Services  (HHS). 

These  agencies  report  no  cost  information  -  not  even  HHS,  whose  report  cover  includes  a  subtitle.  Justification 
of  Estimates  for  Appropriations  Committees." 
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Forward-Looking  Leadership 


9.  Does  the  report  show  how  the  agency's  results  will  make  this  counhy  a  better  place  to  live? 

The  report  should  make  a  solid  case  for  the  agency's  value  and  create  excitement  about  the  challenges  and 
opportunities  ahead  -  regardless  of  actual  past  performance. 

Role  models:  National  Science  Foundation  (NSF)  and  Department  of  Veterans  A  ffairs  (VA). 

NSF's  report  has  a  very  strong  opening:  mission,  exciting  vision,  compelling  message  from  the  director, 
excellent  Executive  Summary  and  interesting  Agency  Profile  and  Examples  of  Achievement.  The  import  of  NSF 
investments  is  supported  by  numerous  highlights  throughout  the  report. 

In  the  VA  report.  Secretary  Principi's  transmittal  letter,  plus  sections  titled  Who  We  Are,  Who  We  Serve  and 
What  We  Accomplished  all  do  a  good  job  of  promising  care  to  veterans  and  reminding  the  rest  of  us  why  the 
department  exists.  They  also  describe  how  the  VA  uses  strategic  planning  to  deliver  lienefits  that  meet  veterans' 
expectations. 

Bad  influences:  Social  Securiti/  Administration  (SSA), 

No  sense  of  mission  emerges  from  this  report.  The  first  paragraph  of  the  report  reads  as  follows: 

"The  Government  Performance  and  Results  Act  of  1993  (GPRA)  requires  Federal  agencies  to  develop  and 
institutionalize  processes  to  plan  for  and  measure  mission  performance.  GPRA  defines  specific  efforts,  at  the 
highest  level,  that  agencies  must  undertake  to  fulfill  this  requirement." 

The  rest  of  the  document  is  just  as  clinical.  Now'here  does  SSA  speak  directly  to  citizens  al'^out  how^  the  agency 
is  being  managed  strategically  in  order  to  improve  lives. 

10.  Does  the  agency  explain  failures  to  achieve  targets  outlined  in  its  strategic  plan? 

If  an  agency  cannot  identify  reasons  for  failure,  it  is  not  learning  from  the  performance  information  contained  in 
the  report.  Wdiile  unrealistic  goals  should  be  revised  downward,  they  should  not  be  changed  to  make  current 
performance  look  better.  Achievable  but  challenging  goals  should  be  the  intention.  Barriers  to  progi'ess  should 
be  labeled  as  such,  and  efforts  should  be  made  to  remove  them. 

Role  models:  ILS.  Agency  for  International  Development  (USAJDf  Department  of  Veterans  Affairs  (VA). 

W^ith  veiy  few’^  exceptions,  USAID  illustrates  failures  just  as  thoroughly  as  successes.  Unfortunately,  it  isn  t  clear 
that  these  belong  to  USAID.  Problems  with  data  and  external  factors  are  all  out  in  the  open,  though. 


At  the  very  front  of  the  VA's  document,  a  one-page  talie  appears  listing  all  29  key  performance  goals.  Two 
columns  under  "W^as  the  Goal  Achieved?"  are  labeled  "Yes"  and  "No."  Oi^ie  could  not  ask  for  more 
sti*aightforward  presentation.  And  since  (unfortunately)  less  than  half  of  them  were  successes,  the  VA  cannot  be 
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accused  of  '^sandbagging/'  In  fact,  VA  had  plenty  of  explaining  to  do;  in  every  case,  the  department  accepts  full 
responsibility  for  missing  targets  and  offers  explanations  and  strategies  for  improvement. 

Bad  influences:  Nuclear  Regulatoiy  Commission  (NRC),  Department  of  Health  and  Human  Services  (HHS). 

NRC  reports  no  failures  because  of  the  way  it  defined  its  goals  -  no  deaths,  no  releases  of  nuclear  materials,  etc. 
Given  this  definition,  "performance  measurement"  consists  of  moving  data  from  mandatory  reports  filed  by 
operators  of  nuclear  plants  to  the  annual  performance  report.  This  is  a  risky  stategy.  If  failures  do  occur,  it  is 
hard  to  imagine  how  this  scheme  (in  the  context  of  other  shortcomings  of  the  report)  will  make  any  explanation 
credible.  A  more  helpful  approach  would  be  to  find  ways  of  assessing  the  risk  of  nuclear  accidents,  track 
changes  in  risk  over  time,  and  identify  what  portion  of  these  changes  could  be  attributed  to  the  NRC's  activities. 

The  reader  has  no  idea  whether  HHS  was  successful  or  not  in  any  area,  since  only  "selected  performance 
stories"  are  told.  For  the  rest,  it  would  be  necessary  to  read  the  13  separate  bureau  reports.  Even  for  the  items 
highlighted  in  the  HHS  report,  it  is  difficult  to  tell  if  targets  were  met. 

11.  Does  the  report  identify  major  management  challenges? 

Most  agencies  have  some  obstacles  that  impair  their  ability  to  achieve  success  in  all  programs  and  activities, 
such  as  obtaining  dean  financial  audits  from  the  Inspector  General.  The  General  Accounting  Office  and  the 
agencies'  own  Inspectors  General  have  identified  many  of  these  challenges.  These  issues  should  be  addressed 
somewhere  in  the  report,  even  if  the  agency  does  not  explicitly  list  them. 

Role  models:  Department  of  Labor  (DOL)  and  Department  of  Veterans  Affairs  (VA). 

DOL  devotes  an  entire  section  of  its  report  to  major  management  challenges.  The  Inspector  General's  top  10 
areas  of  concern  are  identified  and  strategies  for  addressing  each  are  discussed  in  detail.  Problems  identified  by 
GAO  in  its  series  of  "High  Risk"  reports  are  even  mentioned  in  the  Executive  Summary. 

In  the  VA's  report,  a  section  on  Major  Management  Challenges  discusses  the  problems,  the  Inspector  General's 
recommendations,  current  status,  and  adds  the  department's  comments.  Issues  identified  by  GAO  are  given 
similar  treatment.  In  addition,  brief  descriptions  of  the  impact  of  these  management  problems  on  individual 
goal  achievement  are  included  throughout  the  report  (where  applicable). 

Bad  influences:  Social  Security  Adniinistration. 

This  report  contained  no  discussion  of  major  management  challenges  at  all, 

12,  Does  it  describe  changes  in  policies  or  procedures  to  do  better  next  year? 

The  intent  of  the  Government  Performance  and  Results  Act  is  not  just  reporting  for  its  own  sake.  The  legislation 
enunciates  a  different  way  of  managing  federal  agencies  that  holds  them  accountable  for  results  rather  than 
activities.  The  idea  is  to  gather  information  that  informs  future  action  -  and  to  use  it.  Is  it  evident  that 
knowledge  gained  from  this  process  is  actually  being  used  to  make  decisions  on  future  activities  and  priorities? 

Role  models:  Environmental  Protection  Agency  (EPA),  Small  Business  Administratioit  (SEA). 
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The  EPA  does  this  particularly  well  in  its  discussion  of  management  challenges  and  selection  of  performance 
goals  and  measures.  Efforts  appear  to  be  in  earnest.  The  only  thing  missing  is  evidence  that  proposed  activity 
will  produce  the  desired  effect, 

A  section  called  "'Major  Issues  Facing  SBA"  in  the  front  of  its  report  discusses  how  the  agency  is  changing  the 
way  it  does  business.  A  results-based  management  approach  is  evident  in  adoption  of  better  strategic  goals  and 
measures,  new  data  quality  assurance  methods,  investments  in  training  and  modernization,  and  even  proposed 
legislation. 

Bad  influences:  Departmetif'  of  Education. 

This  report  seems  to  have  been  stripped  of  any  reference  to  the  future.  Postponed  publication  of  the  planning 
document  seems  warranted  by  President  Bush's  interest  in  a  swift  change  of  direction.  Unfortunately,  the 
department  does  not  even  bother  to  describe  the  new  concept  or  attempt  to  generate  enthusiasm  and 
anticipation.  Citizens  deserve  to  know  what's  going  on,  and  this  report  completely  misses  the  opportunity  to 
offer  an  explanation. 
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Trends  and  Observations 


IMPROVEMENT  IS  MODEST,  BUT  WIDESPREAD 

We  employed  the  same  criteria  to  assess  the  fiscal  2000  reports  that  we  used  for  the  fiscal  1999  reports.  The  prin¬ 
cipal  difference  is  that  our  expectations  were  higher,  since  agencies  have  had  a  year  to  learn  from  public  and 
congressional  response  to  their  fiscal  1999  reports.  For  example,  an  agency  could  receive  a  high  score  for  our 
first  transparency  criterion  --  making  its  fiscal  1999  report  available  to  the  public  —  if  the  report  could  be  found 
easily  on  the  agency's  web  site.  For  fiscal  2000  reports,  a  perfect  score  required  that  the  agency  make  the  report 
available  in  clearly-labeled  segments  that  are  easy  to  download.  Thus,  an  agency  had  to  improve  the  absolute 
quality  of  its  fiscal  2000  report  in  order  to  receive  the  same  numerical  score  as  it  received  for  its  fiscal  1999 
report.  We  believe  rising  expectations  aie  appjopriate,  because  no  reports  yet  meet  the  standards  for  transparen¬ 
cy  and  disclosure  that  have  been  set  by  the  annual  reports  of  publicly-held  corporations. 

Changes  in  an  agency's  ranking  from  fiscal  1999  to  fiscal  2000  depend  on  two  factors:  the  quality  of  the  agency  s 
report,  and  the  quality  of  all  of  the  other  agencies'  reports.  Improvement  in  an  agency's  ranking  generally 
means  that  the  agency  improved  its  reporting  to  a  greater  extent  than  other  agencies. 

Although  scoring  was  more  stringent  for  fiscal  2000,  the  average  score  improved  by  about  5  percent  over  fiscal 
1999.  This  suggests  that  there  has  been  some  modest  improvement  in  the  overall  quality  of  federal  performance 
reporting.  As  the  chart  below  indicates,  improvement  was  widespread;  the  majority  of  fiscal  2000  reports 
received  better  scores  than  in  fiscal  1999,  in  spite  of  tighter  scoring.  The  quality  of  reports  is  also  somewhat 
more  even  for  fiscal  2000.  In  fiscal  1999,  quality^  dropped  off  sharply  from  the  level  of  the  three  best  reports. 

Scorecard  Score  Distribution 


RANK 
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BIG  IMPROVEMENT  IS  POSSIBLE 


The  two  most-improved  reports  were  those  produced  by  the  National  Science  Foundation  and  the  Department 
of  Justice.  NSF  leapt  from  24th  (last)  place  in  fiscal  1999  to  sixth  place  in  fiscal  2000,  with  a  gain  of  16  points. 
DOJ's  15-point  gain  allowed  it  to  rise  from  21st  place  in  fiscal  1999  to  fifth  place  in  fiscal  2000. 

The  reports  of  both  agencies  showed  the  most  improvement  in  the  areas  where  they  w^ere  weakest  in  fiscal  1999. 
Half  of  NSF's  improvement  came  in  the  foinvard-looking  leadership  categoiy,  suggesting  that  the  foundation  is 
doing  a  much  better  job  of  explaining  how  information  on  past  performance  will  influence  future  plans.  Half  of 
DOJ's  improvement  occurred  on  the  public  benefits  criteria.  Compared  to  fiscal  1999,  DOJ^s  fiscal  2000  report 
was  much  better  at  articulating  how  its  actions  improve  citizens'  lives. 

These  agencies  clearly  focused  on  improving  the  areas  of  their  reports  that  were  least  adequate  in  fiscal  1999.  At 
the  same  time,  they  managed  modest  improvement  in  other  categories  as  well, 

SUBSTANCE  MATTERS  MOST 

Our  evaluation  of  the  fiscal  2000  reports  also  sheds  some  light  on  the  importance  of  format  versus  substance. 
Each  agency  presented  its  report  in  one  of  three  different  ways:  as  a  stand-alone  document,  merged  with  the  fis¬ 
cal  2000  accountability  report,  or  merged  with  the  fiscal  2002  performance  plan.  In  theoiyv  either  ty'pe  of  combi¬ 
nation  might  produce  a  better  report.  Combining  the  performance  report  and  the  accountability^  report  for  the 
same  year  could  provide  better  integration  of  operating  and  financial  information.  Combining  the  performance 
report  with  the  performance  plan  could  create  much  tighter  linkages  between  observed  performance  and  plans 
for  improvement  in  the  future. 

In  practice,  some  agencies  achieved  these  synergies,  but  others  did  not.  As  the  table  show^s,  merely  combining 
the  performance  report  wdth  another  report  does  not  necessarily  lead  to  a  better  report;  high-ranking  and  low- 
ranking  reports  can  be  found  in  each  format.  The  actual  substance  of  the  report  matters  much  more  than 
wdrether  it  is  combined  with  other  documents. 


Stand-alone 

Report 

Rank 

Combined  with  fiscal  2000 
accountability  report 

Rank 

Combined  with  fiscal  2002 
performance  plan 

RanlT^ 

Veterans 

1 

Labor 

4 

DOT 

2 

NSF 

6 

ERA 

6 

USAID 

3 

GSA 

8 

SBA 

8 

Justice 

5 

Treasury 

13 

Energy 

10 

DOI 

12 

State 

14 

hud" 

10 

DOC 

14 

FEMA 

16 

SSA 

19 

Education 

17 

NASA 

23 

OPM 

20 

Defense 

18 

USDA 

24 

NRC 

21 

liHS 

22 

V 

y 
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Research  Team  and  Project  Design 


THE  SCORECARD  RESEARCH  TEAM 

The  scorecard  research  team  is  headed  by  the  Hon,  Maurice  McTigue,  QSO,  a  distinguished  visiting  scholar  at 
the  Mercatus  Center,  and  Dr.  Jerry  Ellig,  a  senior  research  fellow  at  the  Mercatus  Center.  Dr.  Tyler  Cowen,  gen¬ 
eral  director  of  the  Mercatus  Center,  served  as  advisory  director  for  the  study  A  10~member  advisoiy  panel, 
made  up  of  experts  in  public  management,  cc^rporate  strategy,  and  communications,  reviewed  our  evaluations 
and  analysis. 


PROJECT  MANAGEMENT 

The  Honorable  Maurice  McTigue,  QSO 

Director,  Cjoz^ernnnrnt  Accoimtabilih/  Project 

Dr.  Jerry  Ellig 

Senior  Research  Fellow 

Dr.  Tyler  Cowen 

General  Director 


ADVISORY  PANEL 

Mortimer  Downey 

Pri ncipa I  Con s u  1  tan t 
pbConsult 

John  Kamensky 

Director,  Managing  Jbr  Results  Practice 
Pricewa  terhouseCoopers 

Dr.  Thomas  G.  Kessler 

Director 

Denali  Associates 

Dr.  Donald  F.  Kettl 

Professor  of  Public  Affairs 

The  Robert  M.  La  FoUette  School  of  Public  Affairs  and  Political  Science 
University  of  Wisconsin 

Susan  Robinson  King 

Vice  President  Public  Affairs 
Carnegie  Corporation  of  New  York 
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Dr.  Kathryn  E.  Newcomer 

Professor  and  Chair,  Department  of  Public  Administration 
George  Washington  Ihhversiti/ 

Michael  A.  Rosenbaum 

President  &  Chief  Operating  Officer 
Financial  Relations  Board/BSMG  Worldxvide 

Dr.  John  Sacco 

Associate  Professor,  Public  and  International  Affairs 
George  Mason  University 

Virginia  L.  Thomas 

Director,  Executive  Branch  Relations 
Heritage  Foundation 

Chris  Wye 

Director,  Center  for  Improving  Government  Performance 
National  Academy  of  Public  Administration 


RESEARCH  TEAM 
Steve  Richardson 

Associate  Director,  Government  Accountability  Project 
Mercatiis  Center  at  George  sMason  University 

Adolfo  Laurenti 

P/t.D.  Candidate 

Department  of  Econotnics,  George  Mason  University 

Gerry  Williams 

Center  for  Accountability  and  Performance  Fellow,  American  Society  for  Public  Administration 
Formerly  Performance  Planning  Manager,  US,  Department  of  Transportation 

Dr.  Joe  Johnson 

Dorothy  Donnelley  Moller  Research  Felloio 
Mercatiis  Center  at  George  Mason  University 

Dr.  Bertin  Martens 

Visiting  Research  Scholar,  Institute  for  International  Studies,  Stanford  University 
Formerly  Economist,  Evaluation  Unit,  External  Affairs  Department,  European  Commission 
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Weighting  the  Evaluation  Factors 


To  report  the  results  of  this  study  as  transparently  as  possible,  the  researchers  weighted  the  evaluation  factors 
equally  in  calculating  each  agency's  total  score  and  rankings.  Since  the  summary  table  reports  scores  for  all 
three  evaluation  factors  separately  (ti'ansparency,  public  benefit  and  forward-looking  leadership),  readers  who 
believe  that  one  factor  is  more  important  than  others  can  apply  whatever  weights  they  wish  to  the  separate 
scores  and  recalculate  rankings  accordingly. 

Also  in  the  interest  of  transparency,  all  reports  w'ere  evaluated  against  a  common  scale,  even  though  different 
agencv  missions  mav  make  it  inherently  more  difficult  to  develop  result-oriented  goals  and  measures  or  collect 
appropriate  data.  (For  example,  agencies  that  provide  direct  services,  such  as  the  General  Services 
Administration  or  Federal  Emergency  Management  Agency,  may  find  it  easier  to  identify  and  quantify  their 
contributions  than  an  agency  like  the  State  Department.)  A  majority  of  agencies  produced  better  reports  for  fis¬ 
cal  year  2000  than  they  did  for  fiscal  1999,  but  there  is  still  room  for  improvement  in  even  the  best  reports.  It 
will  probably  take  several  years  before  the  inherent  nature  of  agency  missions  becomes  the  binding  constraint 
that  prevents  some  agencies  from  producing  better  performance  reports. 
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About  the  Mercatus  Center 


The  Mercatus  Center  at  George  Mason  University  is  a  research,  education,  and  outreach  organization.  Through 
the  interdisciplinary  study  of  economics,  law,  and  the  humanities,  Mercatus  seeks  to  advance  our  knowledge  of 
how  societies  achieve  freedom,  prosperity,  and  peace. 

Under  the  direction  of  Hoibert  Harris  Professor  of  Economics  Tyler  Cowen,  the  Mercatus  Center  is  at  the  core  of 
a  community  of  students,  scholars,  and  decision-makers.  This  community  works  together  to  build  and  apply  an 
understanding  of  how  individuals  cooperate  through  the  market  and  political  processes. 


About  the  Mercatus  Center’s 
Government  Accountability  Project 


The  Government  Accountability  Project's  goal  is  to  help  improve  government's  funding  and  policy  decisions  by 
making  transparent  the  public  benefits  produced  with  citizens'  resources.  Full  transparency  brings  praise  and 
criticism  of  results  -  and,  eventually,  change  -  based  upon  maximizing  outcomes  and  minimizing  expenditures. 

Since  1997,  the  Mercatus  Center's  scholars  have  used  a  variety  of  tools  to  focus  attention  on  public  benefits  pro¬ 
duced  as  the  most  critical  factor  in  government  decision-making.  When  quality  information  is  available,  effective 
use  of  public  resources  can  be  maximized  by  linking  performance  and  appropriations. 

For  more  information  about  the  Mercatus  Center  and  the  Government  Accountability  Project,  visit  our  adjoining 
websites  (wnvw.mercatus.org  and  www’.GovernmentAccountability.org),  or  you  can  call  us  at  (70j)  993-4930. 


L 


The  analysis,  interpretations  and  conclusions  in  this  study  are  solely  those  of  the  author  and 
research  team;  they  are  not  positions  of  the  Mercatus  Center  or  George  Mason  University. 
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Appendix: 

Evaluation  Summaries 


Department  of  Veterans  Affairs  (VA) 


r 

EVALUATIONSUMMARY 

r' 

FY  2000  rank: 

1 

FY  1999  rank:  3 

J 

Transparency: 

16 

Min.  per  category: 

4 

Public  Benefits: 

15 

Min.  total: 

12 

Leadership: 

18 

Max.  per  category: 

20 

TOTAL: 

49 

Max,  total: 

60 

v_ 

_ J 

TRANSPARENCY 


•  Report  can  be  found  via  several  different  routes  from  VA  home  page,  and  it  is  clearly  labeled 
"Annual  Performance  Report  FY  2000." 

•  Several  sections  in  the  front  of  the  report  help  provide  an  overview  that  adds  perspective  and 
offers  an  alternative  for  readers  not  interested  in  detail. 

•  Methods,  definitions  and  sources  are  provided  for  each  goal,  but  data  quality  is  an  admitted 
problem.  Not  all  data  are  accessible,  and  external  checks  are  limited, 

•  All  key  performance  goals  with  data  are  displayed  in  readable  charts  with  planned  targets  for 
fiscal  2000  and  desired  trends  indicated.  Many  have  five  years  of  data. 


PUBLIC  BENEFITS 


•  Virtually  all  of  the  strategic  goals  and  objectives  are  outcomes. 

•  VAhas  outcome-based  performance  indices  for  chronic  care,  prevention,  and  pensions/ insurance 
-  but  many  other  measures  focus  on  outputs  and  activities. 

•  Since  VA  seldom  measures  veterans'  well-being  directly  and  doesn't  discuss  external  factors  to 
any  significant  extent,  it  is  difficult  to  isolate  the  department's  contribution  to  veterans'  welfare. 

•  Various  sections  show  spending  by  program,  organization,  and  strategic  goal.  VA  is  far  ahead  of 
most  other  agencies  in  reporting  cost  information. 


LEADERSHIP 


•  Secretary's  letter,  plus  sections  on  Who  We  Are,  Who  W e  Serve  and  What  We  Accomplished,  all 
explain  what  the  department  does  for  veterans  and  remind  non-veterans  why  the  department 
exists. 

•  A  chart  lists  29  key  performance  goals  and  states  whether  they  were  achieved.  The  department 
gives  reasons  when  goals  were  not  achieved. 

•  Major  management  challenges  are  discussed  throughout  the  report  and  also  grouped  into  a 
special  section. 

•  Report  describes  means  and  strategies  for  the  future  to  remedy  failures. 
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Department  of  Transportation  (DOT) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

2 

FY  1999  rank:  2 

Transparency: 

16 

Mm.  per  categon/: 

4 

Public  Benefits: 

16 

Min,  total: 

12 

Leadership: 

15 

Max,  per  categori/: 

20 

TOTAL: 

47 

Max.  total: 

60 

v_ 

J 

TRANSPARENCY 


•  A  link  titled  DOT  Performance  Plan/ Report  in  the  middle  of  the  home  page  downloads  the 
entire  report  in  pdf  format*  No  warning  of  size  or  partial  download  options. 

•  Structure,  sequence,  tables,  and  graphs  are  user-friendly.  Only  thing  missing  is  an  overview. 

•  Data  are  selected,  presented,  and  discussed  convincingly,  with  an  extensive  appendix  discussing 
scope,  sources,  limitations,  and  verification.  DOT's  inspector  general  notes  there  are  some 
problems  with  reliability  and  timeliness  of  data. 

•  Report  often  presents  up  to  10  years  of  data  to  aid  in  assessing  trends. 


PUBLIC  BENEFITS 


•  Goals  are  clearly  stated  in  terms  of  outcomes. 

•  Vast  majority  of  measures  are  outcomes. 

•  Agency  sometimes  demonstrates  that  its  actions  are  responsible  for  achievement  of  its  goals,  but 
often  fails  to  do  so. 

•  Budget  information  is  presented  for  each  performance  goal  and  measure,  including  comparison 
of  fiscal  2001  with  fiscal  2002. 


LEADERSHIP 


•  Mission  and  results-oriented  management  are  clear  for  safety  and  mobility^  programs,  but  less 
clear  in  other  areas. 

•  Progress  report  tables  indicate  ''targets  met"  and  "good  trend"  but  have  no  indicator  of  targets 
missed  outright.  Failures  are  explamed,  but  explanations  often  place  the  blame  on  external 
factors. 

•  An  index  indicates  where  major  management  challenges  can  be  found  in  the  report  and  the 
source  Inspector  General,  the  General  Accounting  Office,  or  the  Office  of  Management  and 
Budget.  Challenges  are  integrated  in  a  substantial  way  to  link  them  to  goal  areas  most  affected. 

•  Assessments  are  made  and  future  plans  are  described  in  detail  at  the  performance  goal  level,  but 
not  at  the  strategic  goal  level. 
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U.S.  Agency  for  International  Development  (USAID) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

3 

FY  1999  rank:  1 

J 

Transparency: 

16 

Min,  per  categonj: 

4 

Public  Benefits: 

13 

Min,  total: 

12 

Leadership: 

14 

Max.  per  categori/: 

20 

TOTAL: 

43 

Max,  total: 

60 

TRANSPARENCY 


•  Link  on  agency's  home  page  is  clearly  labeled  and  includes  a  description  of  the  document  and 
SOO-word  overview.  A  Table  of  Contents  indexes  the  report  into  at  least  a  dozen  convenient 
(<200KB)  pdf  downloads. 

•  Generally  readable  and  well  structured. 

•  Abundance  of  data  with  all  sources  mentioned  (including  web  links).  Quality  problems  are 
admitted. 

•  Most  data  document  general  trends  only  marginally  influenced  by  USAID. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  results-oriented;  objectives  would  be  more  results-oriented  if  not  preceded  by 
modifiers  like  '^promote"  and  "encourage." 

•  Indicators  are  all  results,  but  it's  unclear  whether  USAID  influences  the  results. 

•  It  is  not  clear  how  the  agency  actually  conducts  its  business.  The  report  says  good  things  are 
done  by  organizations  which  are  "funded  by"  USAID  without  offering  specifics  (like  how  much 
the  programs  depend  upon  this  agency). 

•  One  table  shows  costs  by  strategic  goal  -  no  mention  of  costs  elsewhere. 


LEADERSHIP  _ _ _ _ _ — 

•  Report  makes  a  good  case  that  the  U.S.  will  be  a  better  place  if  the  rest  of  the  world  has  higher 
standards  of  living.  The  crucial  question  that  isn't  answered  is  whether  USAID  is  affecting  that. 

•  The  report  illustrates  failures  just  as  thoroughly  as  successes. 

•  Plans  to  address  management  challenges  are  discussed  in  some  detail,  but  without  clearly 
defining  the  problems.  It  would  be  helpful  to  see  relevant  recommendations  from  the  agency  s 
inspector  general  and  the  General  Accounting  Office. 

•  Future  plans  (a  section  under  each  strategic  goal)  appear  extensive  but  usually  only  discuss  new 
projects  “  not  how  ongoing  operations  will  be  improved. 
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Department  of  Labor  (DOL) 


EVALUATIONSUMMARY 

r- 

FY  2000  rank; 

4  ^ 

FY  1999  rank:  5 

y 

Transparency: 

15 

Min,  per  categonj: 

4 

Public  Benefits: 

11 

Min,  total: 

12 

Leadership: 

14 

Max.  per  categonj: 

20 

TOTAL: 

40 

Max.  total: 

60 

v 

_ y 

TRANSPARENCY  _ _ 

•  FeaUired  on  DOUs  home  page,  the  report  can't  be  missed.  An  index  offers  about  two  dozen 
separate  sections  for  pdf  download  -  all  <500KB. 

•  Summary,  structure,  language,  headers,  pictures  and  human  interest  stories  all  make  it  easy  to 
understand  what  DOL  is  saying. 

•  Report  has  a  quite  sophisticated  discussion  of  data  problems,  but  DOL's  inspector  general  still 
identifies  data  quality  as  a  major  problem. 

•  Data  are  in  readable  charts,  and  an  cippendix  explains  all  baselines.  It  would  be  helpful  to  have 
more  than  a  few  years'  worth  of  data  to  establish  trends. 


PUBLIC  BENEFITS 


•  Simplicity  and  outcome  orientation  of  the  strategic  goals  would  be  hard  to  improve  upon. 
Headings  for  performance  goals  are  clearer  and  more  outcome-oriented  than  the  goal  statements 
themselves. 

•  Measures  attempt  to  focus  on  outcomes  but  don't  always  succeed.  Many  goals  target 
participation  levels  or  development/implementation  of  a  specific  plan. 

•  Discussion  of  connection  between  DOL  actions  and  statistics  reported  includes  little  discussion  of 
factors  outside  the  agency's  control. 

•  No  linkage  between  costs  and  results.  Report  admits  that  the  department s  two  largest  agencies 
have  not  cooperated  with  the  chief  financial  officer  to  accomplish  this. 


LEADERSHIP  _ _ _ _ _ 

•  Vision  of  DOL's  role  in  worker  preparation,  safety  and  security  is  sound,  but  information  in 
report  is  not  linked  to  show  how  DOL  is  achieving  the  vision. 

•  Except  by  setting  its  goals  low  in  the  first  place  (admitted  in  "Goal  Assessment'  discussions), 
DOL  does  not  seem  afraid  of  admitting  failure. 

•  The  inspector  general's  top  10  major  management  challenges  are  identified  and  strategies  for 
addressing  each  are  discussed  in  detail. 

•  Strategies  for  improving  results  are  found  for  performance  goals  that  were  not  met,  but  outcome 
goal  summaries  only  acknowledge  need  for  continuous  improvement  without  telling  how  that 
will  be  accomplished. 
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Department  of  Justice  (DOJ) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

5 

FY  1999  rank  21 

Transparency: 

12 

Min.  per  category: 

4 

Public  Benefits: 

13 

Min.  total: 

12 

Leadership: 

13 

Max.  per  category: 

20 

TOTAL: 

38 

Max.  total: 

60 

\ 

J 

TRANSPARENCY 


•  Link  on  home  page  labeled  ''Performance  Reports  and  Plans"  leads  to  Table  of  Contents  with 
numerous  sections  for  dowmload  in  pdf. 

•  Format  is  sound  and  uniform,  but  report  suffers  from  excessive  detail  and  lack  of  navigation  aids 
{executive  summary,  tables,  cross-references,  etc.) 

•  All  information  sources  are  internal  and  reliability  of  data  is  not  discussed. 

•  Data  presentation  is  uniform,  though  only  a  few  years  of  data  are  presented. 


PUBLIC  BENEFITS 


•  Goals  and  objectives  are  mostly  outcomes. 

•  Measures  are  related  to  goals  but  mostly  activities. 

•  DOJ  appears  to  carefully  link  actions  with  goals,  so  theoretically,  successes  at  annual  goal  level 
w’ould  add  up  to  achievement  of  objectives  and  strategic  goals.  No  overview  discussion 
demonstrates  that  this  actually  occurred;  readers  must  draw  own  conclusions. 

•  For  each  annual  goal,  there  is  a  table  of  dollar  and  human  resources  for  the  current  year  and  two 
succeeding  fiscal  years  (through  2002).  Cost  figures  are  presented  as  "needs/'  not  an  accounting 
of  expenditures. 


LEADERSHIP  / _ _ _ 

•  Message  from  the  attorney  general  does  a  good  job  of  describing  public  benefits.  The  rest  of  the 
report  provides  good  backgiound  but  buries  results  in  minutia. 

•  Hard  to  determine  how  successful  DO]  was  at  accomplishing  strategic  goals  and  annual  goals. 
Failures  to  achieve  targeted  performance  measures  are  explained. 

•  Report  includes  inspector  general's  description  of  major  management  challenges  and  DOJ's 
responses. 

•  Plenty  of  program  evaluations  are  under  w^ay,  yet  no  change  in  department-wide  approach  is 
apparent.  In  most  cases,  the  automatic  solution  is  to  "do  more,"  and  what  happened  or  did  not 
happen  this  year  does  not  inform  future  plans. 
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Environmental  Protection  Agency  (EPA) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

6 

FY  1999  rank;  11 

TransparencAc 

13 

Mm.  per  cptegori/: 

4 

Public  Benefits: 

9 

Min,  total: 

12 

Leadership: 

15 

Max,  per  categori/: 

20 

TOTAL: 

37 

Max,  total: 

60 

TRANSPARENCY  _ _ _ _ 

•  Report  was  online  in  March^  with  a  clear  explanation  that  it  is  combined  with  Accountability 
Report  and  convenient  separation  into  downloadable  sections. 

•  Well-organized  and  readable,  despite  abundant  use  of  acronyms.  Overview  section  lists  activities 
rather  than  providing  a  comprehensive  picture. 

•  Data  sources  and  quality  are  discussed  in  great  detail;  concern  is  evident  and  efforts  to  improve 
and  make  data  accessible  are  laudable.  EPA  appears  more  optimistic  about  current  data  quality 
than  its  inspector  general  and  the  General  Accounting  Office. 

•  Tiend  data  are  often  shown,  but  many  charts  show'  data  that  do  not  actually  measure  the 
performance  goal.  Goals  are  changed  too  frequently. 


PUBLIC  BENEFITS 


•  Mission  and  strategic  goals  are  outcomes,  but  report  also  contains  numerous  internal 
management  goals  that  describe  processes.  Annual  performance  goals  are  mostly  activity- 
oriented,  not  outcome-oriented, 

•  Measures  are  seldom  clearly  connected  to  outcomes.  Multiple  measures  that  overly  specify 
targets  for  a  single  goal  obscure  whether  goals  w^ere  met. 

•  Despite  sincere  efforts  to  achieve  positive  environmental  outcomes  that  are  described  in  the  text 
of  this  report,  EPA  fails  to  make  a  convincing  case  that  its  actions  were  responsible  for  the  results 
that  are  reported. 

•  The  overview  and  pie  charts  in  each  section  present  some  cost  information,  but  lack  of  outcome 
measures  makes  it  difficult  to  link  costs  wdth  public  benefits. 


LEADERSHIP  ^ _ _ _ 

•  Quality  of  life  issues  are  clearly  the  focus  of  the  strategic  discussion.  This  emphasis  disappears  at 
the  performance  goal  level. 

•  Failures  to  achieve  targets  are  explained  only  for  annual  performance  goals. 

•  Extensive  discussion  of  management  challenges. 

•  For  management  challenges  and  performance  measures,  prospective  changes  for  the  future  are 
discussed  extensively. 
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National  Science  Foundation  (NSF) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

6 

FY 1999  rank  24 

J 

Transparency: 

13 

Min,  per  categonj: 

4 

Public  Benefits: 

10 

Min.  total: 

12 

Leadership: 

14 

Max.  per  aitegori/: 

20 

TOTAL: 

37 

Max,  total: 

60 

\ 

j 

TRANSPARENCY 


•  The  report  is  posted  online  in  pdf  format  but  not  easy  to  find. 

•  Language,  layout,  and  format  make  the  report  easy  to  understand,  with  just  a  few  unexplained 
technical  terms. 

•  Procedures  for  collecting  and  assessing  data  are  discussed  in  detail  and  have  been  reviewed  by 
an  external  party,  PricewaterhouseCoopers. 

•  Trends  are  difficult  to  identify  because  only  a  few  years  of  data  are  presented  and  outcomes  are 
evaluated  only  as  "'successful"  or  "not  successful." 


PUBLIC  BENEFITS 


•  Strategic  goals  are  outcomes;  goals  for  supporting  management  and  investment  processes  are 
outputs. 

•  Performance  measures  are  usually  outcomes  if  the  goal  is  an  outcome,  and  outputs  if  the  goal  is 
an  output. 

•  It  is  not  clear  what  opportunities  for  discoveries  would  have  been  forgone  but  for  NSF  support. 

•  No  link  between  goals/ results  and  costs. 


LEADERSHIP  _ _ _ _ _ 

•  Mission  and  vision  enunciate  the  potential  for  NSF  to  improve  our  quality  of  life. 

•  Discussions  of  success  and  failure  are  frank  and  thorough;  only  drawbacks  are  a  few  cases  where 
unconvincing  explanations  offered  for  low'ering  goals. 

•  Management  challenges  are  listed  but  discussed  only  briefly. 

•  In  each  goal  area,  plans  for  improvement  are  shared  that  indicate  strategic  thinking  and  earnest 
effort  to  put  the  Government  Performance  and  Results  Act  to  good  use. 
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General  Services  Administration 


r 

EVALUATIONSUMMARY 

FY  2000  rank: 

8 

FY  1999  rank:  9 

Transparenc}^: 

10 

Mm.  fcr  categon/: 

4 

Public  Benefits: 

14 

Min,  total: 

12 

Leadership: 

12 

Max.  per  categonj: 

20 

TOTAL: 

36 

Max.  total: 

60 

v_ 

TRANSPARENCY 


•  Report  not  found  on  the  GSA  web  site. 

•  Report  is  in  four  pieces  and  not  well  integrated,  making  it  hard  to  assess  GSA's  overall 
performance. 

•  Information  on  data  sources  and  methodolog}"  is  good,  especially  for  the  Public  Buildings 
Service.  Federal  Supply  Service  provides  a  helpful  discussion  of  data  validity. 

•  Data  for  several  years  are  included,  though  inconsistently. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  not  outcomes  for  the  public,  but  internal  management  goals  for  operating 
divisions. 

•  Performance  measures  are  largely  appropriate,  given  the  goals. 

•  At  the  performance  goal  level,  there  is  a  clear  connection  between  GSA  s  activities  and  the 
savings  to  its  U.S.  government  customers, 

•  Goals  and  measures  all  revolve  around  fiscal  responsibility  and  efficiency.  Despite  failure  of  the 
agency  to  highlight  its  own  cost  of  operations,  one  cannot  read  this  report  without  obtaining  a 
fair  picture  of  the  costs  and  benefits  of  its  activities. 


LEADERSHIP  ^ _ _ _ _ 

•  Benefits  to  the  general  public  seem  to  be  assumed,  not  stated.  The  separate  p^rts  of  GSAseem  to 
know  what  they  are  doing  and  can  demonstrate  achievement. 

•  Targets  appear  to  have  been  reachable  but  not  easy,  and  failures  are  explained. 

•  Appendix  summarizes  challenges  noted  by  the  inspector  general.  General  Accounting  Office,  and 
Congress,  and  it  responds  to  them.  Presentation  appears  to  minimize  importance,  though, 
because  it  does  not  link  these  issues  to  the  agency's  mission. 

•  Prospective  changes  for  next  year  are  explained  adec|uately  at  the  performance  goal  level,  but 
not  at  the  organization-wide  level. 
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Small  Business  Administration  (SBA) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

8 

FY  1999  rank:  9 

J 

Transparency; 

10 

Min,  per  categon/: 

4 

Public  Benefits: 

12 

Min,  total: 

12 

Leadership: 

14 

Ma,x,  per  categori/: 

20 

TOTAL: 

36 

Max,  total: 

60 

TRANSPARENCY 


•  Report  not  found  on  SBA  website. 

•  Layout  and  format  sometimes  make  the  report  hard  to  follow. 

•  Data  Validity  section  discusses  strateg^v  a  pilot  project  to  test  relevancy  and  quality,  and  program 
evaluations.  No  sources  are  named  ~  even  for  the  few  data  (national  statistics  on  businesses)  that 
originate  outside  the  agency. 

•  Display  of  data  in  tabular  format  is  not  that  helpful  for  visualizing  trends,  and  data  are  too 
focused  on  process  details  to  lend  context. 


PUBLIC  BENEFITS 


•  New  strategic  goals  and  strategies  are  expressed  as  outcomes,  but  it  is  not  clear  how  they  are 
related  to  older  items  called  "outcomes"  in  the  rest  of  the  report. 

•  Virtually  all  performance  measures  assess  whether  the  right  amount  of  money  got  to  people  - 
outputs.  Statements  are  more  results-based  than  the  actual  measures. 

•  Intentions  and  strategies  are  very  good.  Significant  goal  changes  have  been  made,  and  they  are 
in  the  right  direction.  In  the  meantime,  though,  results  do  not  support  the  SBA's  success  claims, 

•  Goals  and  results  are  linked  to  costs  at  the  performance  indicator  level. 


LEADERSHIP  _ 

•  Agency  appears  competent  but  focused  on  a  single  constituency  and  traditional  processes  instead 
of  benefits  to  the  American  public. 

•  Failures  are  admitted  and  explained  at  the  performance  goal  level.  At  the  strategic  goal  level,  it 
is  hard  to  tell  if  SBA  was  successful,  or  not. 

•  A  15-page  section  includes  auditors'  recommendations,  actions  taken  and  actions  planned 
(including  target  dates). 

•  Results-based  management  approach  is  evident  in  adoption  of  new  goals  and  measures,  data 
quality  assurance  methods,  investments  in  training  and  modernization,  and  even  proposed 
legislation. 
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Department  of  Housing  and  Urban  Development  (HUD) 


EVALUATIONSUMMARY 

A 

FY  2000  rank: 

10 

V _ 

FY  1999  rank:  13 

Transparenc)^: 

10 

Min.  per  categor}/: 

4 

Public  Benefits: 

13 

Min.  total: 

12 

Leadership: 

12 

Max.  per  category: 

20 

TOTAL: 

35 

Max.  total: 

60 

V _ y 


•  Report  not  found  on  the  HUD  web  site. 

•  Organization  and  language  are  clear  and  simple,  generally.  One  source  of  confusion  is  the  habit 
this  agency  has  formed  of  referring  to  its  programs  in  terms  of  the  legal  codes  that  created  them  - 
e.g.  Section  221  (d)  (4). 

•  For  outcome  measures,  source  agencies  are  listed,  and  a  footnote  says  to  consult  the  Annual 
Performance  Plan  for  sources  of  other  data.  This  is  not  adequate,  since  there  isn't  a  statement  of 
any  kind  as  to  the  quality  of  said  data. 

•  Graphs  present  trend  data  for  most  indicators  for  at  least  three  years.  Graphs  need  explanation 
to  render  them  more  meaningful. 


PUBLIC  BENEFITS 


•  Strategic  goals,  objectives,  and  outcome  measures  all  focus  on  results. 

•  Agency's  contribution  toward  its  target  population's  well-being  is  not  clearly  established.  HUD 
appears  aware  of  this  deficiency  and  is  trying  to  correct  it. 

•  No  systematic  link  of  results  with  costs. 


•  Direction  of  change  (more  strategic,  results-based  approach)  and  attitude  toward  accountability 
are  encouraging.  Ability  to  deliver  on  promises  and  degree  of  commitment  to  implementation  is 
not  emphasized. 

•  Explanations  of  failures  are  dear  and  adequate  at  the  objective  and  indicator  levels.  At  the 
strategic  goal  level  and  occasionally  for  objectives,  it  is  difficult  to  tell  if  HUD  is  successful  or  not. 

•  Management  challenges  identified  by  the  inspector  general  are  discussed.  No  discussion  of 
management  issues  raised  by  the  General  Accounting  Office  in  its  "High  Risk"  report. 

•  Tliis  report  focuses  on  past  activities.  Admission  of  numerous  external  factors  and  unknowns 
would  logically  indicate  need  for  program  evaluations,  but  no  such  plans  are  evident. 
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Department  of  Energy  (DOE) 


EVALUATIONSUMMARY 


FY  2000  rank: 

10 

X 

FY1999ratik:  14 

Transparenc}^^ 

10 

Mifh  per  categon/: 

4 

Public  Benefits: 

12 

Min.  total: 

12 

Leadership: 

13 

Max.  per  categori/: 

20 

TOTAL: 

35 

Max.  total: 

60 

TRANSPARENCY 


•  This  report  is  posted  on  DOE's  website,  but  it  is  not  easy  to  find.  The  link  is  labeled 
"Performance  and  Accountability  Report"-  a  single  SMB  pdf  file. 

•  This  report  is  well  structured;  its  style  is  clear  and  easy  to  read.  Acronyms  are  always  defined. 

•  Selection  of  quantifiable  measures  makes  verification  straightforward;  these  (internal)  procedures 
are  discussed.  No  information  on  data  sources  and  quality. 

•  By  articulating  its  measures  in  the  form  of  a  "to-do  list,"  DOE  renders  year-to-year  comparison 
extremely  difficult. 


PUBLIC  BENEFITS 


•  Intention  to  focus  goals  and  objectives  on  outcomes  is  clear,  though  they  are  often  expressed  as 
activities,  such  as  "promote,"  "foster"  and  "strive." 

•  Performance  goals  are  complex  and  unique  to  the  fiscal  year. 

•  Link  between  actions  and  accomplishment  of  goals  is  clear,  but  only  because  goals  are  stated  as 
activities  rather  than  outcomes, 

•  The  amount  of  cost  information  and  progress  in  mapping  out  a  performance  budgeting  scheme  is 
quite  impressive. 


X 

LEADERSHIP  _ _ _ _ _ 

•  DOE  appears  focused  more  on  its  own  processes  than  on  a  central  mission  that  emphasizes 
public  benefits. 

•  In  the  overview,  each  discussion  of  objectives  summarizes  relevant  successes  and  failures  and 
closes  with  a  frank  self-assessment.  In  the  appendix,  the  Results  section  explains  what  happened 
and  is  followed  by  an  action  plan. 

•  The  report  mentions  major  management  challenges  briefly. 

•  A  sincere  desire  to  clean  up  management  messes  and  e\ddence  of  willingness  to  change  appear 
thi'oughout.  The  various  plans  aren't  held  together  by  any  comprehensive  strategy,  though. 
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Department  of  the  Interior  (DOl) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

12 

FY 1999  rank;  11 

Transparency^: 

12 

Min,  per  categor}/: 

4 

Public  Benefits: 

9 

Min,  total: 

12 

Leadership: 

12 

Max,  per  category: 

20 

TOTAL: 

33 

Max.  total: 

60 

L 

j 

TRANSPARENCY 


•  Not  found  on  the  DOI  web  site. 

•  Well-organized  and  generally  easy  to  read. 

•  Data  validation  and  verification  ''matrix"  (in  pilot  stage),  developed  in  concert  with  the  IG, 
appears  excellent.  Most  sources  and  data  checks  are  internal 

•  Actual  and  planned  figures  are  shown  for  3-5  years  for  "key"  performance  measures. 


PUBLIC  BENEFITS 


•  Most  goals  are  outcome-oriented,  though  many  are  ambiguous  and  could  be  interpreted  as 
activities, 

•  Most  performance  measures  are  activities,  not  outcomes. 

•  Connection  between  DOFs  actions  and  outcomes  is  discussed  only  briefly 

•  Minimal  discussion  of  costs,  though  the  report  says  DOI  is  "beginning  to  capture  costs  at  the  goal 
level." 


LEADERSHIP 


•  Report  enunciates  laudable  goals  but  little  vision  of  how  DOI  will  achieve  them. 

•  Failures  to  achieve  annual  goals  are  explained,  but  discussion  of  failure  at  any  other  level  is 
either  confusing  or  absent. 

•  Management  Issues  section  identifies  major  management  challenges,  states  which  bureau(s)  are 
responsible  for  addressing  each  one,  and  identifies  relevant  performance  measures. 

•  Desire  to  improve  management  and  to  become  more  accountable  appears  sincere.  Overall, 
though,  the  long-standing  management  challenges  seem  to  have  overwhelmed  the  department  s 
ability  to  achieve  a  clear  focus  on  results. 
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Department  of  the  Treasury  (Treasury) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

13 

FY  1999  rank:  5 

j 

Transparency: 

10 

Mm.  per  ciitegory: 

Public  Benefits: 

12 

Min,  total: 

Leadership: 

10 

Max.  per  cafegorif: 

TOTAL: 

32 

Max.  total: 

L 

j 

TRANSPARENCY 


•  The  report  is  posted  on  the  department's  website  but  quite  difficult  to  locate, 

•  Organization  and  language  are  fine,  but  it  reads  like  a  database  with  approximately  300 
performance  measures, 

•  Limited  discussion  of  data  reliability.  Report  claims  data  are  sufficiently  accurate,  but  Tieasui'y  s 
inspector  general  disagrees. 

•  Charts  include  several  years  of  data  but  say  little  about  Treasury's  performance. 


PUBLIC  BENEFITS 


•  Most  goals  and  objectives  are  outcomes. 

•  Vast  majority  of  performance  measures  focus  on  processes,  but  the  connection  to  outcomes  is 
usually  explained. 

•  Actions  and  results  are  linked  at  the  objective  level  but  not  at  higher  levels,  such  as  mission  and 
strategic  goals. 

•  Costs  are  reported  only  by  mission  and  not  linked  to  strategic  objectives,  strategic  goals,  or 
results. 


LEADERSHIP 


•  Contribution  of  Treasury  to  public  welfare  is  clear  only  because  it  can  be  inferred  easily  from 
goals;  no  overview  makes  this  link  explicitly. 

•  Failures  to  achieve  performance  goals  are  discussed  adequately.  Failures  to  achieve  taigets  foi 
mission  and  strategic  goals  are  not  explained  even  though  most  of  them  were  not  met. 

•  An  appendix  lists  management  challenges  identified  by  the  inspector  general  and  General 
Accounting  Office  and  explains  how  the  department  is  addressing  them. 

•  This  is  a  very  backward-looking  report  with  no  discussion  of  plans  to  improve  performance  in 
the  future. 
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Department  of  State  (State) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

14  ^ 

FY 1999  rank:  17 

Transparency: 

11 

Mm.  per  aitegorij: 

4 

Public  Benefits: 

9 

Min,  total: 

12 

Leadership: 

10 

Max.  per  cafegori/: 

20 

TOTAL: 

30 

Max.  total: 

60 

_ Y 

TRANSPARENCY 


•  Program  Performance  Report  link  on  home  brings  up  description  of  the  report  and  sepaiate 
sections  for  download  in,  pdf, 

•  Text  is  easy  to  read  and  informative,  but  would  benefit  from  headings  and  formatting  that  would 
guide  the  reader  more  easily  through  it. 

•  Data  sources  are  given,  but  not  very  useful  to  outsiders  because  reference  is  made  only  to 
bureaus  and  their  internal  data  or  reporting  systems, 

•  Tiends  are  difficult  to  identify  because  few  outcome  measures  are  defined  in  ways  that  facilitate 
year-to-year  comparison. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  mostly  outcomes;  performance  goals  are  mostly  outputs, 

•  Our  researchers  had  difficulty  identifying  performance  indicators,  their  relationships  to 
performance  goals,  outcomes  and  strategic  goals,  and  why  they  would  matter  to  anyone  outside 
the  department. 

•  Report  claims  successes  without  explaining  why  State  is  responsible  for  them. 

•  No  discussion  of  costs. 


LEADERSHIP  _ _ _ _ _ _ _ - 

•  Brief  history  and  description  of  the  department,  national  interest  areas  and  strategic  goals  make 
the  role  of  State  and  its  link  to  citizens'  well-being  clear.  On  the  other  hand,  the  mission 
statement  is  focused  inward  and  activity-oriented. 

•  Use  of  "partially  successful"  assessments  obviated  the  need  to  admit  failure  in  90  percent  of  the 
cases  where  targets  w^ere  not  met.  State  admits  not  meeting  targets  but  generally  ought  to  be 
more  introspective  and  critical  in  evaluating  outcomes, 

•  Major  management  challenges  are  briefly  mentioned,  without  discussion  of  how  long  the 
problems  have  existed  or  progress  in  addressing  them. 

•  No  indication  of  how”  the  information  in  this  report  will  be  used  to  improve  State  s  performance 

in  the  future. 
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Department  of  Commerce  (DOC) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

14 

FY  1999  rank:  22 

Transparenc}^' 

9 

Mm.  per  categon/: 

4 

Public  Benefits: 

10 

Min.  total: 

12 

Leadership: 

11 

Max.  per  category: 

20 

TOTAL: 

30 

Max.  total: 

60 

L 

TRANSPARENCY 


•  The  report  is  not  on  DOC's  website. 

•  Essential  information  is  easily  found.  Opportunities  for  improvement:  somewhat  technical 
presentation  in  spots,  excessive  detail,  insufficient  graphics  and  stray  comments  or  terms  that 
are  undefined  or  unexplained. 

•  Most  bureaus  describe  data  verification  and  validation  procedures  without  assessing  reliability  of 
data.  Secretary's  foreword  acknowledges  data  weaknesses. 

•  So  many  measures  are  new  that  there  is  little  trend  data.  Reasons  for  changes  can  be  ascertained 
only  with  great  effort. 


PUBLIC  BENEFITS 


•  Strategic  goals  and  objectives  are  related  to  outcomes  but  stated  as  outputs  (provide,  support, 
promote,  observe).  Performance  goals  are  mostly  outputs. 

•  Performance  measures  at  bureau  level  are  a  mix  of  outcomes  and  c^utputs. 

♦  No  information  on  whether  outcomes  are  improving  in  the  three  major  mission  areas,  or  whethei 
the  department  is  responsible  for  any  improvement. 

♦  Each  bureau  presents  a  four-year  budget  histoiy,  portrayed  as  resource  requirements,  foi  each 

of  its  performance  goals. 


LEADERSHIP 


•  A  highlights  section  lists  some  attractive  outcomes  for  citizens  that  are  not  reflected  in 
departmental  goals  or  objectives. 

•  Failures  to  achieve  performance  goals  are  thoroughly  explained,  but  the  explanations  often  fail  to 
identify  core  problems. 

•  Most  bureaus  discuss  major  management  challenges,  with  varying  levels  of  quality;  some 
interpret  the  term  simply  as  a  chance  to  ask  for  more  resources. 

•  Department's  desire  to  improve  implementation  of  the  Government  Performance  and  Results  Act 
is  evident.  But  it  is  not  clear  that  a  commitment  exists  outside  these  documents. 
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Federal  Emergency  Management  Agency  (FEMA) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

16 

FY  1999  rank;  17 

j 

Transparency^; 

10 

M/n.  per  categori/: 

4 

Public  Benefits: 

8 

Min.  total: 

12 

Leadership: 

11 

Max.  per  category: 

20 

TOTAL: 

29 

Max.  total: 

60 

L. 

J 

TRANSPARENCY 


•  This  report  could  be  easier  to  find,  but  it  is  on  FEMA's  website  in  pdf  format. 

•  Layout,  writing  style  and  use  of  graphics  combine  to  make  this  report  very  easy  to  read.  It  needs 
a  table  of  contents,  executive  summary,  and  clearer  labels  for  goals. 

♦  Director's  letter  apologizes  for  poor  data  quality.  No  discussion  of  this  elsewhere. 

♦  Baseline  and  trend  data  are  supplied  where  possible,  along  with  explanations  when  performance 
goals  have  changed. 


PUBLIC  BENEFITS  ^ _ 

•  Strategic  goats  are  outcomes.  Performance  goals  are  mostly  activities. 

•  Most  indicators  are  activity-oriented. 

•  Internal  focus  of  goals  and  indicators  hampers  connection  of  activities  with  results.  Links 
between  achievements  and  indicators  are  often  unclear. 

•  No  discussion  of  costs. 


LEADERSHIP 


•  Introduction  provides  excellent  explanation  of  how  FEMA  makes  this  counhy  a  better  place  to 
live.  Rest  of  report  provides  occasional  glimpses. 

•  Failures  are  explained  when  they  can  be  identified.  It  is  often  hard  to  identify  failures  from  the 
information  presented. 

•  Discussion  of  management  challenges  appears  otitdated,  since  it  references  challenges  identified 
in  fiscal  2000  Annual  Performance  Plan  a  document  now  two  years  old. 

•  Discussion  of  plans  for  improvement,  in  the  context  of  specific  activities,  is  scattered  throughout 
the  report. 
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Department  of  Education  (ED) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

17 

FY 1999  rank:  4 

Transparency”: 

7 

h4in,  per  categori/: 

4 

Public  Benefits: 

11 

MiiL  total: 

12 

Leadership: 

10 

Max,  per  categonj: 

20 

TOTAL: 

28 

J 

Max,  total: 

60 

\ 

TRANSPARENCY 


•  Report  not  found  on  ED  web  site. 

•  Organization  is  confusing;  it  is  divided  into  two  volumes  that  are  not  integrated* 

•  It  is  labeled  an  "interim"  report  because  President  BtislYs  education  proposals  could  substantially 
alter  the  department's  activities  -  but  it  also  omits  most  fiscal  2000  results. 

•  Few’^  data  are  presented,  data  quality  is  flagged  as  a  major  problem,  and  no  solution  is  discussed. 

•  Where  data  are  presented,  trends  through  fiscal  1999  and  goals  for  fiscal  2000  are  well 
documented. 


PUBLIC  BENEFITS 


•  Three  of  four  strategic  goals  are  outcomes,  as  are  the  strategic  objectives. 

•  Performance  measures  are  a  mix  of  outcomes  and  outputs. 

•  Agency  fails  to  show  it  has  made  a  difference  --  in  part  because  it  reports  few  results. 

•  Funding  history  is  provided  by  program  but  not  linked  to  goals. 


LEADERSHIP 


•  Goals  and  objectives  are  results  most  Americans  w''Ouid  view  as  desirable,  but  failure  to  report 
results  instills  little  confidence  that  the  department  is  making  a  difference. 

•  Failures  to  meet  targets  are  explained  thoroughly. .  .when  failures  can  be  identified. 

•  A  25-page  section  in  Volume  TI  identifies  four  areas  of  concern  expressed  by  the  inspector  general 
and  General  Accounting  Office,  It  furnishes  background  information,  lists  strategies,  and 
assesses  progress  on  each.  Missing  are  specific  recommendations  and  plans  for  the  future. 

•  This  report  seems  to  have  been  stripped  of  any  reference  to  the  future. 
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Department  of  Defense  (DOD) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

18  ^ 

FY  1999  rank:  7 

J 

Transparency: 

12 

Min,  per  categonj: 

4 

Public  Benefits: 

8 

Min,  total: 

12 

Leadership: 

7 

Max,  per  categori/: 

20 

TOTAL: 

27 

Max,  total: 

60 

TRANSPARENCY  _ _ _ 

♦  Released  late.  Listed  on  DOD's  web  page  under  ''Reports''  as  the  "Government  Performance  and 
Results  Act  -  DOD  FY  2000  Performance  Report." 

•  Easy  to  read,  but  acronyms  are  overused  and  it  needs  a  more  thorough  overview. 

*  Data  verification  and  validation  is  discussed  for  each  measure.  All  checks  are  internal,  making  it 
harder  for  outsiders  to  verify  data.  Most  discussion  focuses  on  methods  of  gathering  data  rather 
than  assessment  of  quality. 

•  Fiscal  1998  baselines  and  fiscal  2001  targets  bracketed  actual  data  for  the  last  two  years  and 
targets  for  the  current  year  in  tables. 


PUBLIC  BENEFITS 


•  Corporate  Goals  and  Annual  Performance  Goals  are  all  stated  as  activities.  Two  goals  -<  readiness 
and  capability  are  somewhat  outcome-oriented. 

•  Most  performance  measures  are  activities. 

•  Outcomes  for  the  public  are  never  defined;  nowhere  is  actual  performance  in  militaiy  action 
mentioned. 

•  Report  briefly  describes  numerous  cost  measures  related  to  acquisition,  infrastructure,  and 
financial  management.  No  discussion  of  cost-effectiveness. 


LEADERSHIP 


•  Document  reads  like  a  piece  of  internal  reporting,  not  an  attempt  to  explain  to  citizens  how  DOD 
improves  their  lives. 

•  Failures  to  meet  goals  are  noted  and  explained,  but  there  is  little  critical  assessment  of  strategies 
or  actions. 

•  A  table  lists  performance  indicators  corresponding  to  management  challenges  identified  by  the 
department's  inspector  general  and  the  General  Accounting  Office.  There  is  no  discussion  of 
w^hat  the  department  is  doing  to  address  the  challenges. 

•  Rarely  describes  changes  to  improve  performance  next  year.  A  surprising  number  of  factors  are 
considered  outside  DOD's  control,  including  personnel  retention. 
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Social  Security  Administration  (SSA) 


r 

EVALUATIONSUMMARY 

r 

FY  2000  rank: 

19 

FY  1999  rank:  8 

Transparent^: 

11 

Min,  per  categonj: 

4 

Public  Benefits: 

8 

Min,  total: 

12 

Leadership: 

7 

Max.  per  category: 

20 

TOTAL: 

26 

Max.  total: 

so 

TRANSPARENCY 


•  Posted  on  SSA's  web  site.  It  is  a  chapter  in  the  Accountability  Report. 

•  Language  is  readable,  but  meaning  is  more  difficult  to  extract  due  to  goal  structure  and 
formatting. 

•  No  sources  listed  for  data;  reader  is  simply  assured  that  data  collection  is  automated  and  reports 
are  subject  to  review. 

•  Multiple  years  of  data  were  presented  for  most  goals.  Some  goals  involve  completion  of 
specified  activities,  so  there  is  no  opportunity  to  examine  ti'ends. 


PUBLIC  BENEFITS 


•  The  first  three  strategic  goals  are  outcomes  but  the  other  two  are  not.  Objectives  are  input  or 
output-oriented  (e.g.  providing  information,  making  payments  on  time  or  having  a  well-trained 
workforce), 

•  Almost  all  measures  focus  on  activities,  and  many  are  meaningless  to  people  outside  the  agency. 

•  SSAdoesnT  demonstrate  effectiveness  of  its  programs  -  impact  on  ''economic  well-being  of  the 
public,"  New  barometer  measures  are  a  step  in  that  direction. 

•  No  useful  information  linking  goals  and  results  to  costs. 


LEADERSHIP 


•  Nowhere  does  SSA  speak  directly  to  citizens  about  how  the  agency  is  being  managed 
strategically  in  order  to  improve  their  lives. 

•  Some  unmet  targets  are  explained  thoroughly,  but  others  are  not. 

•  No  discussion  of  major  management  challenges. 

•  No  discussion  of  strategies  for  the  future. 
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Office  of  Personnel  Management  (OPM) 


r 

EVALUATIONSUMMARY 

r- 

FY  2000  rank: 

20  ^ 

FY  1999  rank;  14 

J 

Transparency: 

9 

Min,  per  categonj: 

4 

Public  Benefits: 

6 

Min.  total: 

12 

Leadership: 

10 

Max.  per  categonj: 

20 

TOTAL: 

25 

Max.  total: 

60 

L. 

y 

TRANSPARENCY 


•  The  report  is  not  available  on  OPM's  website. 

•  Readability  is  good  -  language  is  plain  and  acronyms  are  explained.  Structure,  though 
(especially  sheer  length),  inhibits  understanding. 

•  Some  data  sources  are  offered.  Most  are  internal,  however,  and  the  inspector  general  reports  that 
barely  half  are  reliable  indicator's.  Known  weaknesses  are  discussed  and  a  new  measurement 
framework  is  under  development. 

•  Measures  are  still  very  new,  so  data  are  limited  in  many  cases  to  baselines.  Effort  to  present 
emerging  trends  is  commendable. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  clearly  activities  and  read  more  like  policy  statements  whose  outcomes  are 
vague. 

•  Aside  from  a  few  survey-based  measures,  all  performance  measures  focus  on  activities, 
hidicators  are  too  numerous  (458)  to  be  meaningful  in  a  single  performance  report. 

•  It  is  difficult  to  isolate  the  contribution  of  OPM  to  its  "ultimate  objective"  ~  "...that  the  Federal 
workforce  be  citizen-centered,  results  oriented,  efficient  and  effective." 

•  No  cost  data  linked  to  goals  or  results. 


LEADERSHIP 


•  Director's  letter  and  a  management  section  attempt  to  offer  a  vision  of  how^  OPM  benefits 
Americans,  but  this  is  unsupported  by  the  rest  of  this  activity-focused  report. 

•  Most  failures  are  blamed  on  insufficient  resources. 

•  Some  management  challenges  are  discussed. 

•  Some  plans  for  the  future  are  discussed,  but  it's  not  clear  how  they  reflect  what  the  agency  has 
learned  from  past  performance. 
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Nuclear  Regulatory  Commission  (NRC) 


EVALUATIONSUMMARY 

FY  2000  rank: 

21 

■A 

FY  1999  rank:  17 

Transparency: 

8 

Mm.  per  category: 

4 

Public  Benefits: 

8 

Min,  total: 

12 

Leadership: 

6 

Max,  per  category: 

20 

TOTAL: 

22 

Max,  total: 

60 

\ 

J 

•  The  report  is  online  but  difficult  to  locate.  A  link  to  "Planning  and  Financial  Management  brings 
up  a  list  of  budget  documents,  one  of  which  is  "NEC's  Accountability  Report/'  a  pdf  download, 

•  The  report  is  well  organized  and  easy  to  follow, 

•  Extensive  discussion  of  daia  verification  procedures  and  reliability;  data  sources  are  not 
disclosed. 

•  Hardly  any  meaningful  trend  data  are  reported,  even  though  licensees  surely  must  have  reported 
much  of  the  data  before  1999. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  ou  tcomes,  but  basically  limited  to  avoiding  disaster. 

•  Performance  measures  focus  on  occurrence  of  nuclear  incidents.  This  approach  captures  nothing 
about  how  risks  are  changing  over  time,  and  by  itself  it  tells  us  little  about  the  agency's,  rather 
than  facility  operators',  performance. 

•  The  agency  conveys  an  image  of  responsible  enforcement.  But  due  to  very  poor  measurement 
choices,  they  fail  to  convince  readers  that  safety  is  greater  as  a  direct  result  of  their  activities. 

•  Costs  not  linked  to  performance  or  outcomes. 


LEADERSHIP  _ 

•  While  the  report  shows  there  were  no  significant  nuclear  accidents,  it  shares  no  overall  strategy 
or  specific  actions  that  can  attribute  this  result  to  the  NRC, 

•  NRC  goals  of  no  deaths,  no  releases,  etc.  are  an  easy  way  out,  reducing  the  performance 
measurement  task  to  simpN  moving  data  from  mandatory  operator  reports  to  the  annual 
performance  report. 

•  A  summary  of  the  inspector  general's  assessment  in  the  appendix  is  inadequate,  because 
problems  aren't  really  defined. 

•  NRC  seems  to  be  changing  a  few  systems.  Report  reveals  no  strategic  thinking. 
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Department  of  Health  and  Human  Services  (HHS) 


EVALUATIONSUMMARY 

r" 

FY  2000  rank: 

22 

FY  1999  rank:  20 

Transparency: 

7 

Min.  per  cafegonj: 

4 

Public  Benefits: 

8 

Min.  total: 

12 

Leadership: 

6 

Max.  per  categon/: 

20 

TOTAL: 

21 

Max.  total: 

60 

V _ y 


TRANSPARENCY 


•  Report  not  found  on  HHS  web  site, 

•  The  document  ineffectively  combines  the  fiscal  2002  performance  plan  and  the  fiscal  2000 
performance  report.  The  first  section,  called  a  "'user  guide,"'  is  a  50-page  treatise  on  GPRAthat 
contains  no  overview  of  goals,  strategies,  or  results. 

•  Half  of  HHS  has  not  yet  even  reported  all  fiscal  1999  performance  goal  data. 

•  No  trend  data  are  presented,  and  most  fiscal  2000  data  are  unavailable. 


PUBLIC  BENEFITS 


•  Strategic  goals  are  mostly  outcomes,  "Selected"  performance  goals  included  in  this  report  are  a 
mix  of  outcomes  and  outputs. 

•  HHS  has  950  annual  performance  goals.  It  reports  on  a  couple  hundred  of  them  in  this  summary 
without  offering  any  rationale  or  methodology  for  making  its  selections.  Some  are  outputs  and 
some  are  outcomes. 

•  No  causal  link  is  demonstrated  between  HHS  activities  and  outcome  goals. 

•  A  label  on  the  cover  reads  "Justification  of  Estimates  for  Appropriations  Committees/'  even 
though  report  includes  no  discussion  of  service  delivery  costs. 


LEADERSHIP  ^ _ _ _ 

•  Departmental  goals  are  laudable.  This  report  fails  to  inspire,  though;  there  is  no  department- 
wide  strateg)^  and  no  overview  of  results. 

•  Only  "selected  performance  stories"  are  told  ~  making  it  very  difficult  to  judge  either  success  or 
failure.  Others  are  supposedly  presented  in  the  13  separate  bureau  reports. 

•  Management  and  Coordination  section  mentions  the  inspector  general's  and  General  Accounting 
Office's  attention  to  this  matter  without  identifying  the  challenges  -  even  though  12  of  the  14  are 
deemed  "serious,"  Instead,  a  half-dozen  are  selected  and  presented  without  discussion  of  their 
tlrreats  to  performance  or  current  status. 

•  It  isn't  clear  w^hat  w^orked  and  w^hat  did  not  last  year.  No  strategy  or  commitment  evident  for 
improving  performance. 
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National  Aeronautics  and  Space  Administration  (NASA) 


r 

EVALUATIONSUMMARY 

FY  2000  rank: 

23  ^ 

FY 1999  rank:  14 

Transparency: 

5 

Mm.  per  category: 

4 

Public  Benefits: 

6 

Min,  total: 

12 

Leadership: 

7 

Max.  per  categonj: 

20 

TOTAL: 

18 

Max.  total: 

60 

j 

TRANSPARENCY 


•  Report  not  found  on  NASA  web  site. 

•  Organization  is  confusing  (enterprises  and  cross-cutting  processes),  summary  tables  are  hard  to 
decipher,  technical  language  is  excessive,  use  of  acronyms  is  tiresome,  measures  are  too 
numerous  and  editing  is  poor. 

•  Reliability  of  data  is  suspect.  The  inspector  general  conducted  audits  only  on  "selected 
measures,  the  Advisory  Council  review  of  agency  performance  is  considered  part  of  the  data 
verification  and  validation  process,  and  sources  and  reliability  checks  are  all  internal. 

•  Tiends  cannot  be  discerned  because  most  measures  take  the  form  of  activity  lists. 


PUBLIC  BENEFITS 


•  At  the  organization-wide  level,  NASA  has  not  stated  goals  at  all;  instead,  it  has  identified 
"enterprises"  and  "processes"  -  each  of  which  contains  numerous  goals  and  objectives. 

•  Most  performance  measures  are  activity -based,  usually  reading  like  a  "to  do"  list. 

•  It  is  ver\^  difficult  to  discern  links  between  activities  and  the  agency’^'s  ultimate  objectives.  Some 
progress  is  evident,  and  since  NASA  has  few  competitors  in  the  field,  it  is  reasonable  to  credit 
them  with  whatever  results  are  obtained.  This  also  begs  the  question,  however,  of  what  is 
achievable. 

•  No  link  between  goals,  results,  and  costs. 


LEADERSHIP 


•  Though  NASA  arguably  has  the  most  exciting  and  visionary  job  of  any  federal  agency,  its  report 
fails  to  communicate  the  benefits  of  NASA's  activities. 

•  Failures  at  the  performance  target  level  are  explained,  but  strategic  failures  are  most  often 
blamed  on  external  factors. 

•  International  Space  Station  is  the  only  major  management  challenge  discussed. 

•  No  convincing  discussion  of  how  NASA  plans  to  improve  in  the  future. 
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Department  of  Agriculture  (USDA) 


r 

EVALUATIONSUMMARY 

r~ 

FY  2000  rank: 

24 

FY  1999  rank:  22 

Transparency: 

N/A 

Min.  per  categonj: 

4 

Public  Benefits: 

N/A 

Min.  total: 

12 

Leadership: 

N/A 

Max.  per  categonj: 

20 

TOTAL: 

N/A 

J 

Max.  total: 

60 

NO  REPORT  WAS  AVAILABLE  AS  OF  ^27/01. 
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A  .  Report  Title  :2"'^  Annual  Performance  Report  Scorecard:  Which 
Federal  Agencies  Inform  the  Public 


Downloaded  From  the  Internet  16  May  2001 

C.  Report's  Point  of  Contact:  (Mercatus  Center  George  Mason 
University  Fairfax  Va  22030 

D.  Currently  Applicable  Classification  Level;  Unclassified 


E.  Distribution  Statement  A:  Approved  for  Public  Release 

F.  The  foregoing  information  was  compiled  and  provided  by: 

DTIC-OCA,  Initials; _ JC _ Preparation  Date:  2001/09/27 


The  foregoing  information  should  exactly  correspond  to  the  Title,  Report  Number,  and  the  Date  on 
the  accompanying  report  document.  If  there  are  mismatches,  or  other  questions,  contact  the 
above  OCA  Representative  for  resolution. 


